[jira] [Comment Edited] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531434#comment-17531434
 ] 

John Zhuge edited comment on HADOOP-16159 at 5/3/22 11:33 PM:
--

Here is the deadlock in SPARK-39094:

thread-1
{noformat}
t0: URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(conf))
when conf is not initialized. i.e. `properties` is null
t1: conf.get(...) ->
conf.getProps() acquires Configuration object ->
conf.loadResources ->
any code that acquires some sort of JAVA lock, e.g. 
`URL.streamHandlerLock`{noformat}
thread-2
{noformat}
t2: random Java code e.g. JceSecurity static initializer ->
new URL(): acquire the same URL.streamHandlerLock ->
URL.getURLStreamHandler ->
    FsUrlStreamHandlerFactory.createURLStreamHandler ->
    FileSystem.getFileSystemClass ->
    Configuration.getClass -> ...
    Configuration.getProps acquires Configuration object{noformat}
 


was (Author: jzhuge):
thread-1
{noformat}
t0: URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(conf))
when conf is not initialized. i.e. `properties` is null
t1: conf.get(...) ->
conf.getProps() acquires Configuration object ->
conf.loadResources ->
any code that acquires some sort of JAVA lock, e.g. 
`URL.streamHandlerLock`{noformat}

thread-2
{noformat}
t2: random Java code e.g. JceSecurity static initializer ->
new URL(): acquire the same URL.streamHandlerLock ->
URL.getURLStreamHandler ->
    FsUrlStreamHandlerFactory.createURLStreamHandler ->
    FileSystem.getFileSystemClass ->
    Configuration.getClass -> ...
    Configuration.getProps acquires Configuration object{noformat}
 

> Deadlock when using FsUrlStreamHandlerFactory
> -
>
> Key: HADOOP-16159
> URL: https://issues.apache.org/jira/browse/HADOOP-16159
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.5, 3.1.2
>Reporter: Ajith S
>Priority: Critical
>
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
>  This induces the thread lock
> Thread 1 : Does load class which will do below
>  - waiting to lock <0x0005c0c1e5e0> (a 
> org.apache.hadoop.conf.Configuration)
> at 
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:420)
> at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)  
> Thread 2 : Create new URL
>  - waiting to lock <0x0005b7991168> (a 
> org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source)
> at 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown 
> Source)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>  - locked <0x0005c0c1e5e0> (a 

[jira] [Comment Edited] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531437#comment-17531437
 ] 

John Zhuge edited comment on HADOOP-16159 at 5/3/22 10:29 PM:
--

I think HADOOP-15331 actually fixed this issue because it ensures the cloned 
Configuration is initialized with a call of getProps() while 
`FsUrlStreamHandlerFactory(conf)` clones the conf.
{code:java}
public Configuration(Configuration other) {
  synchronized(other) {
// Make sure we clone a finalized state
// Resources like input streams can be processed only once
other.getProps();
this.resources = (ArrayList) other.resources.clone();
if (other.properties != null) {
  this.properties = (Properties)other.properties.clone();
}
 {code}
Can we have another pair of eyes?


was (Author: jzhuge):
I think HADOOP-15331 actually fixed this issue because it ensures the cloned 
Configuration is initialized with a call of getProps()
{code:java}
public Configuration(Configuration other) {
  synchronized(other) {
// Make sure we clone a finalized state
// Resources like input streams can be processed only once
other.getProps();
this.resources = (ArrayList) other.resources.clone();
if (other.properties != null) {
  this.properties = (Properties)other.properties.clone();
}
 {code}
Can we have another pair of eyes?

> Deadlock when using FsUrlStreamHandlerFactory
> -
>
> Key: HADOOP-16159
> URL: https://issues.apache.org/jira/browse/HADOOP-16159
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.5, 3.1.2
>Reporter: Ajith S
>Priority: Critical
>
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
>  This induces the thread lock
> Thread 1 : Does load class which will do below
>  - waiting to lock <0x0005c0c1e5e0> (a 
> org.apache.hadoop.conf.Configuration)
> at 
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:420)
> at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)  
> Thread 2 : Create new URL
>  - waiting to lock <0x0005b7991168> (a 
> org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source)
> at 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown 
> Source)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>  - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at 

[jira] [Commented] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531437#comment-17531437
 ] 

John Zhuge commented on HADOOP-16159:
-

I think HADOOP-15331 actually fixed this issue because it ensures the cloned 
Configuration is initialized with a call of getProps()
{code:java}
public Configuration(Configuration other) {
  synchronized(other) {
// Make sure we clone a finalized state
// Resources like input streams can be processed only once
other.getProps();
this.resources = (ArrayList) other.resources.clone();
if (other.properties != null) {
  this.properties = (Properties)other.properties.clone();
}
 {code}
Can we have another pair of eyes?

> Deadlock when using FsUrlStreamHandlerFactory
> -
>
> Key: HADOOP-16159
> URL: https://issues.apache.org/jira/browse/HADOOP-16159
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.5, 3.1.2
>Reporter: Ajith S
>Priority: Critical
>
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
>  This induces the thread lock
> Thread 1 : Does load class which will do below
>  - waiting to lock <0x0005c0c1e5e0> (a 
> org.apache.hadoop.conf.Configuration)
> at 
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:420)
> at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)  
> Thread 2 : Create new URL
>  - waiting to lock <0x0005b7991168> (a 
> org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source)
> at 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown 
> Source)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>  - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:599)
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory

[John Zhuge (Jira)]

[ https://issues.apache.org/jira/browse/HADOOP-16159 ]


John Zhuge deleted comment on HADOOP-16159:
-

was (Author: jzhuge):
Can we fix it by replacing `conf` with `this.conf` in this constructor?
{code:java}
public FsUrlStreamHandlerFactory(Configuration conf) {
  this.conf = new Configuration(conf);
  // force init of FileSystem code to avoid HADOOP-9041
  try {
FileSystem.getFileSystemClass("file", conf); === replace with 
this.conf
  } catch (IOException io) {
throw new RuntimeException(io);
  }
  this.handler = new FsUrlStreamHandler(this.conf);
} {code}
In this way, no other thread can lock this private Configuration object.

> Deadlock when using FsUrlStreamHandlerFactory
> -
>
> Key: HADOOP-16159
> URL: https://issues.apache.org/jira/browse/HADOOP-16159
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.5, 3.1.2
>Reporter: Ajith S
>Priority: Critical
>
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
>  This induces the thread lock
> Thread 1 : Does load class which will do below
>  - waiting to lock <0x0005c0c1e5e0> (a 
> org.apache.hadoop.conf.Configuration)
> at 
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:420)
> at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)  
> Thread 2 : Create new URL
>  - waiting to lock <0x0005b7991168> (a 
> org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source)
> at 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown 
> Source)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>  - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:599)
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531435#comment-17531435
 ] 

John Zhuge edited comment on HADOOP-16159 at 5/3/22 10:19 PM:
--

Can we fix it by replacing `conf` with `this.conf` in this constructor?
{code:java}
public FsUrlStreamHandlerFactory(Configuration conf) {
  this.conf = new Configuration(conf);
  // force init of FileSystem code to avoid HADOOP-9041
  try {
FileSystem.getFileSystemClass("file", conf); === replace with 
this.conf
  } catch (IOException io) {
throw new RuntimeException(io);
  }
  this.handler = new FsUrlStreamHandler(this.conf);
} {code}
In this way, no other thread can lock this private Configuration object.


was (Author: jzhuge):
Is this a possible fix by replacing `conf` with `this.conf` in the following 
code?
{code:java}
public FsUrlStreamHandlerFactory(Configuration conf) {
  this.conf = new Configuration(conf);
  // force init of FileSystem code to avoid HADOOP-9041
  try {
FileSystem.getFileSystemClass("file", conf); === replace with 
this.conf
  } catch (IOException io) {
throw new RuntimeException(io);
  }
  this.handler = new FsUrlStreamHandler(this.conf);
} {code}
In this way, no other thread can lock this private Configuration object.

> Deadlock when using FsUrlStreamHandlerFactory
> -
>
> Key: HADOOP-16159
> URL: https://issues.apache.org/jira/browse/HADOOP-16159
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.5, 3.1.2
>Reporter: Ajith S
>Priority: Critical
>
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
>  This induces the thread lock
> Thread 1 : Does load class which will do below
>  - waiting to lock <0x0005c0c1e5e0> (a 
> org.apache.hadoop.conf.Configuration)
> at 
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:420)
> at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)  
> Thread 2 : Create new URL
>  - waiting to lock <0x0005b7991168> (a 
> org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source)
> at 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown 
> Source)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>  - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> 

[jira] [Commented] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531435#comment-17531435
 ] 

John Zhuge commented on HADOOP-16159:
-

Is this a possible fix by replacing `conf` with `this.conf` in the following 
code?
{code:java}
public FsUrlStreamHandlerFactory(Configuration conf) {
  this.conf = new Configuration(conf);
  // force init of FileSystem code to avoid HADOOP-9041
  try {
FileSystem.getFileSystemClass("file", conf); === replace with 
this.conf
  } catch (IOException io) {
throw new RuntimeException(io);
  }
  this.handler = new FsUrlStreamHandler(this.conf);
} {code}
In this way, no other thread can lock this private Configuration object.

> Deadlock when using FsUrlStreamHandlerFactory
> -
>
> Key: HADOOP-16159
> URL: https://issues.apache.org/jira/browse/HADOOP-16159
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.5, 3.1.2
>Reporter: Ajith S
>Priority: Critical
>
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
>  This induces the thread lock
> Thread 1 : Does load class which will do below
>  - waiting to lock <0x0005c0c1e5e0> (a 
> org.apache.hadoop.conf.Configuration)
> at 
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:420)
> at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)  
> Thread 2 : Create new URL
>  - waiting to lock <0x0005b7991168> (a 
> org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source)
> at 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown 
> Source)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>  - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:599)
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531434#comment-17531434
 ] 

John Zhuge commented on HADOOP-16159:
-

thread-1
{noformat}
t0: URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(conf))
when conf is not initialized. i.e. `properties` is null
t1: conf.get(...) ->
conf.getProps() acquires Configuration object ->
conf.loadResources ->
any code that acquires some sort of JAVA lock, e.g. 
`URL.streamHandlerLock`{noformat}

thread-2
{noformat}
t2: random Java code e.g. JceSecurity static initializer ->
new URL(): acquire the same URL.streamHandlerLock ->
URL.getURLStreamHandler ->
    FsUrlStreamHandlerFactory.createURLStreamHandler ->
    FileSystem.getFileSystemClass ->
    Configuration.getClass -> ...
    Configuration.getProps acquires Configuration object{noformat}
 

> Deadlock when using FsUrlStreamHandlerFactory
> -
>
> Key: HADOOP-16159
> URL: https://issues.apache.org/jira/browse/HADOOP-16159
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.5, 3.1.2
>Reporter: Ajith S
>Priority: Critical
>
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
>  This induces the thread lock
> Thread 1 : Does load class which will do below
>  - waiting to lock <0x0005c0c1e5e0> (a 
> org.apache.hadoop.conf.Configuration)
> at 
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:420)
> at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
> at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)  
> Thread 2 : Create new URL
>  - waiting to lock <0x0005b7991168> (a 
> org.apache.spark.util.MutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.parsers.DOMParser.(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source)
> at 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown 
> Source)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>  - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
> at 
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:599)
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-18132) S3A to support exponential backoff when throttled

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-18132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17495352#comment-17495352
 ] 

John Zhuge edited comment on HADOOP-18132 at 2/21/22, 2:49 PM:
---

S3A already performs retries with exponential backoff on certain S3 errors. For 
details, please check out 
[https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Retry_and_Recovery].


was (Author: jzhuge):
S3A already performs retries on S3 errors. For details, please check out 
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Retry_and_Recovery.

> S3A to support exponential backoff when throttled
> -
>
> Key: HADOOP-18132
> URL: https://issues.apache.org/jira/browse/HADOOP-18132
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Reporter: Holden Karau
>Priority: Major
>
> S3 API has limits which we can exceed when using a large number of 
> writers/readers/or listers. We should add randomized-exponential back-off to 
> the s3 client when it encounters:
>  
> com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request 
> rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; 
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-18132) S3 exponential backoff

[John Zhuge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge resolved HADOOP-18132.
-
Resolution: Not A Problem

S3A already performs retries on S3 errors. For details, please check out 
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Retry_and_Recovery.

> S3 exponential backoff
> --
>
> Key: HADOOP-18132
> URL: https://issues.apache.org/jira/browse/HADOOP-18132
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Reporter: Holden Karau
>Priority: Major
>
> S3 API has limits which we can exceed when using a large number of 
> writers/readers/or listers. We should add randomized-exponential back-off to 
> the s3 client when it encounters:
>  
> com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request 
> rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; 
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15633:

Fix Version/s: 3.1.2

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, 
> HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> at 

[jira] [Updated] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature

[John Zhuge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14206:

Fix Version/s: (was: 3.0.4)
   2.10.1
   2.9.3

> TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
> -
>
> Key: HADOOP-14206
> URL: https://issues.apache.org/jira/browse/HADOOP-14206
> Project: Hadoop Common
>  Issue Type: Test
>  Components: fs, test
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: Jim Brennan
>Priority: Major
> Fix For: 3.3.0, 2.9.3, 3.1.4, 3.2.2, 2.10.1
>
> Attachments: HADOOP-14206-branch-2.10.001.patch, 
> HADOOP-14206.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt:
> {noformat}
> Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem
> testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem)  Time elapsed: 
> 0.19 sec  <<< ERROR!
> java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: 
> java.security.SignatureException: Invalid encoding for signature
>   at com.jcraft.jsch.Session.connect(Session.java:565)
>   at com.jcraft.jsch.Session.connect(Session.java:183)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: 

[jira] [Comment Edited] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054495#comment-17054495
 ] 

John Zhuge edited comment on HADOOP-14206 at 3/8/20, 6:45 PM:
--

Pushed to trunk, branch-3.2, branch-3.1, branch-2.10, and branch-2.9. Thanks 
[~Jim_Brennan] for the contribution!


was (Author: jzhuge):
Pushed to trunk, branch-3, branch-3.2, branch-3.1, branch-2.10, and branch-2.9. 
Thanks [~Jim_Brennan] for the contribution!

> TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
> -
>
> Key: HADOOP-14206
> URL: https://issues.apache.org/jira/browse/HADOOP-14206
> Project: Hadoop Common
>  Issue Type: Test
>  Components: fs, test
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: Jim Brennan
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-14206-branch-2.10.001.patch, 
> HADOOP-14206.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt:
> {noformat}
> Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem
> testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem)  Time elapsed: 
> 0.19 sec  <<< ERROR!
> java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: 
> java.security.SignatureException: Invalid encoding for signature
>   at com.jcraft.jsch.Session.connect(Session.java:565)
>   at com.jcraft.jsch.Session.connect(Session.java:183)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> 

[jira] [Commented] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054495#comment-17054495
 ] 

John Zhuge commented on HADOOP-14206:
-

Pushed to trunk, branch-3, branch-3.2, branch-3.1, branch-2.10, and branch-2.9. 
Thanks [~Jim_Brennan] for the contribution!

> TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
> -
>
> Key: HADOOP-14206
> URL: https://issues.apache.org/jira/browse/HADOOP-14206
> Project: Hadoop Common
>  Issue Type: Test
>  Components: fs, test
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: Jim Brennan
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-14206-branch-2.10.001.patch, 
> HADOOP-14206.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt:
> {noformat}
> Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem
> testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem)  Time elapsed: 
> 0.19 sec  <<< ERROR!
> java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: 
> java.security.SignatureException: Invalid encoding for signature
>   at com.jcraft.jsch.Session.connect(Session.java:565)
>   at com.jcraft.jsch.Session.connect(Session.java:183)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature

[John Zhuge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14206:

Fix Version/s: 3.2.2
   3.1.4
   3.3.0
   3.0.4
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
> -
>
> Key: HADOOP-14206
> URL: https://issues.apache.org/jira/browse/HADOOP-14206
> Project: Hadoop Common
>  Issue Type: Test
>  Components: fs, test
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: Jim Brennan
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-14206-branch-2.10.001.patch, 
> HADOOP-14206.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt:
> {noformat}
> Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem
> testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem)  Time elapsed: 
> 0.19 sec  <<< ERROR!
> java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: 
> java.security.SignatureException: Invalid encoding for signature
>   at com.jcraft.jsch.Session.connect(Session.java:565)
>   at com.jcraft.jsch.Session.connect(Session.java:183)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053605#comment-17053605
 ] 

John Zhuge commented on HADOOP-14206:
-

Give a few days for others to chime in, then I will commit.

> TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
> -
>
> Key: HADOOP-14206
> URL: https://issues.apache.org/jira/browse/HADOOP-14206
> Project: Hadoop Common
>  Issue Type: Test
>  Components: fs, test
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: Jim Brennan
>Priority: Major
> Attachments: HADOOP-14206-branch-2.10.001.patch, 
> HADOOP-14206.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt:
> {noformat}
> Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem
> testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem)  Time elapsed: 
> 0.19 sec  <<< ERROR!
> java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: 
> java.security.SignatureException: Invalid encoding for signature
>   at com.jcraft.jsch.Session.connect(Session.java:565)
>   at com.jcraft.jsch.Session.connect(Session.java:183)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

[jira] [Commented] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052596#comment-17052596
 ] 

John Zhuge commented on HADOOP-14206:
-

[~Jim_Brennan] Thanks for finding the fix and submitting a patch!

+1 LGTM

> TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
> -
>
> Key: HADOOP-14206
> URL: https://issues.apache.org/jira/browse/HADOOP-14206
> Project: Hadoop Common
>  Issue Type: Test
>  Components: fs, test
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: Jim Brennan
>Priority: Major
> Attachments: HADOOP-14206-branch-2.10.001.patch, 
> HADOOP-14206.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt:
> {noformat}
> Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem
> testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem)  Time elapsed: 
> 0.19 sec  <<< ERROR!
> java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: 
> java.security.SignatureException: Invalid encoding for signature
>   at com.jcraft.jsch.Session.connect(Session.java:565)
>   at com.jcraft.jsch.Session.connect(Session.java:183)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>   at 
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
>   at 
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
>   at 
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: 

[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032831#comment-17032831
 ] 

John Zhuge commented on HADOOP-12990:
-

Yes indeed. Actually 3 apps (lz4 tool, Hadoop, and Spark) claiming the `.lz4` 
extension.

> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268366#comment-15268366
 ] 

John Zhuge edited comment on HADOOP-12990 at 2/8/20 6:36 AM:
-

[~cavanaug], lz4 command line and hadoop-lz4 use the same lz4 codec library. 
The difference is only the framing, see my comment and hack on 4/3.

Questions for your use case:
 * Do your JSON files contain a single JSON object or many JSON records?
 * After ingesting into HDFS, how do you plan to use the data?
 * Have you considered these splittable container file formats with 
compression: SequenceFile, RCFile, ORC, Avro, Parquet? In the container, they 
can choose any Hadoop codec, including LZ4.


was (Author: jzhuge):
[~cavanaug], lz4 command line and hadoop-lz4 use the same lz4 codec library. 
The difference is only the framing, see my comment and hack on 4/3.

Questions for your use case:
* Do your JSON files contain a single JSON object or many JSON records?
* After ingesting into HDFS, how do you plan to use the data?
* Have considered these splittable container file formats with compression: 
SequenceFile, RCFile, ORC, Avro, Parquet? In the container, they can choose any 
Hadoop codec, including LZ4.


> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032199#comment-17032199
 ] 

John Zhuge edited comment on HADOOP-12990 at 2/8/20 6:31 AM:
-

OOM usually indicates format mismatch, e.g., reading a large block size, then 
trying to allocate memory.

After looking into Spark code, I realized I was wrong about using Hadoop codec. 
Spark uses its own LZ4 codec based on 
[lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 
2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124].

Its javadoc points out:
{quote} * @note The wire protocol for this codec is not guaranteed to be 
compatible across versions
 * of Spark. This is intended for use as an internal compression utility within 
a single Spark
 * application.{quote}
Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 
tool.

Your best bet may be writing a Java decompression application with a compatible 
version of lz4-java, e.g., 1.4.0 used by Spark 2.3.


was (Author: jzhuge):
OOM usually indicates format mismatch, e.g., reading a large block size, then 
trying to allocate memory.

After looking into Spark code, I realized I was wrong about using Hadoop codec. 
Spark uses its own LZ4 codec based on 
[lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 
2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124].

Its javadoc points out:
{quote} * @note The wire protocol for this codec is not guaranteed to be 
compatible across versions
 * of Spark. This is intended for use as an internal compression utility within 
a single Spark
 * application.{quote}
Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 
tool.

Your best bet may be writing a Java decompression application with a compatible 
version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0.

> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 

[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032202#comment-17032202
 ] 

John Zhuge commented on HADOOP-12990:
-

[~redriver] since Hadoop is not involved, you might want to file a Spark JIRA 
and continue the discussion there.

> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032199#comment-17032199
 ] 

John Zhuge edited comment on HADOOP-12990 at 2/7/20 8:35 AM:
-

OOM usually indicates format mismatch, e.g., reading a large block size, then 
trying to allocate memory.

After looking into Spark code, I realized I was wrong about using Hadoop codec. 
Spark uses its own LZ4 codec based on 
[lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 
2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124].

Its javadoc points out:
{quote} * @note The wire protocol for this codec is not guaranteed to be 
compatible across versions
 * of Spark. This is intended for use as an internal compression utility within 
a single Spark
 * application.{quote}
Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 
tool.

Your best bet may be writing a Java decompression application with a compatible 
version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0.


was (Author: jzhuge):
OOM usually indicates format mismatch, e.g., reading a large block size, then 
trying to allocate memory.

After looking into Spark code, I realized I was wrong about using Hadoop codec. 
Spark uses its own LZ4 codec base on 
[lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 
2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124].

Its javadoc points out:
{quote} * @note The wire protocol for this codec is not guaranteed to be 
compatible across versions
 * of Spark. This is intended for use as an internal compression utility within 
a single Spark
 * application.{quote}
Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 
tool.

Your best bet may be writing a Java decompression program with a matching or 
compatible version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0.

> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 

[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032199#comment-17032199
 ] 

John Zhuge commented on HADOOP-12990:
-

OOM usually indicates format mismatch, e.g., reading a large block size, then 
trying to allocate memory.

After looking into Spark code, I realized I was wrong about using Hadoop codec. 
Spark uses its own LZ4 codec base on 
[lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 
2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124].

Its javadoc points out:
{quote} * @note The wire protocol for this codec is not guaranteed to be 
compatible across versions
 * of Spark. This is intended for use as an internal compression utility within 
a single Spark
 * application.{quote}
Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 
tool.

Your best bet may be writing a Java decompression program with a matching or 
compatible version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0.

> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031853#comment-17031853
 ] 

John Zhuge commented on HADOOP-12990:
-

[~redriver] Thanks for the info.
{quote}But it is still confusing for me how did Spark achieves this,
{quote}
What Spark achieved? Write and/or read? Spark probably uses Hadoop codec on top 
of my head.

> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-14038) Rename ADLS credential properties

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006221#comment-17006221
 ] 

John Zhuge edited comment on HADOOP-14038 at 12/31/19 7:57 PM:
---

[~hvanhovell] Before AdlConfKeys.addDeprecatedKeys is called in AdlFileSystem 
static initializer, somebody might have already created a Configuration 
instance and called `get` on a deprecated property, thus we need to reload 
configurations to apply the ADLS deprecations. Please note ADLS properties live 
in core-site. The cases may be extremely rare.


was (Author: jzhuge):
Before AdlConfKeys.addDeprecatedKeys is called in AdlFileSystem static 
initializer, somebody might have already created a Configuration instance and 
called `get` on a deprecated property, thus we need to reload configurations to 
apply the ADLS deprecations. Please note ADLS properties live in core-site. The 
cases may be extremely rare.

> Rename ADLS credential properties
> -
>
> Key: HADOOP-14038
> URL: https://issues.apache.org/jira/browse/HADOOP-14038
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14038.001.patch, HADOOP-14038.002.patch, 
> HADOOP-14038.003.patch, HADOOP-14038.004.patch, HADOOP-14038.005.patch, 
> HADOOP-14038.006.patch, HADOOP-14038.007.patch
>
>
> Rename properties with prefix dfs.adls. to fs.adl.
> Rename adl.dfs.enable.client.latency.tracker to 
> adl.enable.client.latency.tracker



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14038) Rename ADLS credential properties

[John Zhuge (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006221#comment-17006221
 ] 

John Zhuge commented on HADOOP-14038:
-

Before AdlConfKeys.addDeprecatedKeys is called in AdlFileSystem static 
initializer, somebody might have already created a Configuration instance and 
called `get` on a deprecated property, thus we need to reload configurations to 
apply the ADLS deprecations. Please note ADLS properties live in core-site. The 
cases may be extremely rare.

> Rename ADLS credential properties
> -
>
> Key: HADOOP-14038
> URL: https://issues.apache.org/jira/browse/HADOOP-14038
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14038.001.patch, HADOOP-14038.002.patch, 
> HADOOP-14038.003.patch, HADOOP-14038.004.patch, HADOOP-14038.005.patch, 
> HADOOP-14038.006.patch, HADOOP-14038.007.patch
>
>
> Rename properties with prefix dfs.adls. to fs.adl.
> Rename adl.dfs.enable.client.latency.tracker to 
> adl.enable.client.latency.tracker



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863683#comment-16863683
 ] 

John Zhuge commented on HADOOP-16112:
-

[~leosun08], thanks for finding and reporting the issue! I can take a look.

Just echo [~hexiaoqiao]'s comment, the new unit test passes without any fix, is 
it valid? I understand race condition is hard to reproduce.

BTW, have you or your users hit this issue often? What was the impact other 
than being surprised by the unexpected trash location?

> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
>  if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " + 
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
>  // find the path which is not a directory, and modify baseTrashPath
>  // & trashPath, then mkdirs
>  Path existsFilePath = baseTrashPath;
>  while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff}  other thread deletes existsFilePath here ，the results 
> doesn't  meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff}   there is 
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff}   when delete /user/u_sunlisheng/b/a. if existsFilePath is 
> deleted, the result is 
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff}  so  when existsFilePath is deleted, don't modify 
> baseTrashPath.    {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
>  existsFilePath.toString(), existsFilePath.toString() + Time.now())
>  );
> trashPath = new Path(baseTrashPath, trashPath.getName());
>  // retry, ignore current failure
>  --i;
>  continue;
>  } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e; 
> break; }



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16219) Hadoop branch-2 to set java language version to 1.8

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804234#comment-16804234
 ] 

John Zhuge commented on HADOOP-16219:
-

+1

On Thu, Mar 28, 2019 at 12:33 PM Steve Loughran (JIRA) 


-- 
John


> Hadoop branch-2 to set java language version to 1.8
> ---
>
> Key: HADOOP-16219
> URL: https://issues.apache.org/jira/browse/HADOOP-16219
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.10.0
>Reporter: Steve Loughran
>Priority: Major
>
> Java 7 is long EOL; having branch-2 require it is simply making the release 
> process a pain (we aren't building, testing, or releasing on java 7 JVMs any 
> more, are we?). 
> Staying on java 7 complicates backporting, JAR updates for CVEs (hello 
> Guava!)  are becoming impossible.
> Proposed: increment javac.version = 1.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684427#comment-16684427
 ] 

John Zhuge commented on HADOOP-12125:
-

Working on it [~medb].

> Retrying UnknownHostException on a proxy does not actually retry hostname 
> resolution
> 
>
> Key: HADOOP-12125
> URL: https://issues.apache.org/jira/browse/HADOOP-12125
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Reporter: Jason Lowe
>Assignee: John Zhuge
>Priority: Major
>
> When RetryInvocationHandler attempts to retry an UnknownHostException the 
> hostname fails to be resolved again.  The InetSocketAddress in the 
> ConnectionId has cached the fact that the hostname is unresolvable, and when 
> the proxy tries to setup a new Connection object with that ConnectionId it 
> checks if the (cached) resolution result is unresolved and immediately throws.
> The end result is we sleep and retry for no benefit.  The hostname resolution 
> is never attempted again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610935#comment-16610935
 ] 

John Zhuge commented on HADOOP-12125:
-

My colleague has a fix for this issue.

> Retrying UnknownHostException on a proxy does not actually retry hostname 
> resolution
> 
>
> Key: HADOOP-12125
> URL: https://issues.apache.org/jira/browse/HADOOP-12125
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Reporter: Jason Lowe
>Assignee: John Zhuge
>Priority: Major
>
> When RetryInvocationHandler attempts to retry an UnknownHostException the 
> hostname fails to be resolved again.  The InetSocketAddress in the 
> ConnectionId has cached the fact that the hostname is unresolvable, and when 
> the proxy tries to setup a new Connection object with that ConnectionId it 
> checks if the (cached) resolution result is unresolved and immediately throws.
> The end result is we sleep and retry for no benefit.  The hostname resolution 
> is never attempted again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14710) Uber-JIRA: Support AWS Snowball

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610081#comment-16610081
 ] 

John Zhuge commented on HADOOP-14710:
-

[~kdzhao], I have added you as a contributor, just in case you'd like to take 
on this or any JIRA.

> Uber-JIRA: Support AWS Snowball
> ---
>
> Key: HADOOP-14710
> URL: https://issues.apache.org/jira/browse/HADOOP-14710
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Major
>
> Support data transfer between Hadoop and [AWS 
> Snowball|http://docs.aws.amazon.com/snowball/latest/ug/whatissnowball.html].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14710) Uber-JIRA: Support AWS Snowball

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610080#comment-16610080
 ] 

John Zhuge commented on HADOOP-14710:
-

Thanks for the investigation [~kdzhao]

> Uber-JIRA: Support AWS Snowball
> ---
>
> Key: HADOOP-14710
> URL: https://issues.apache.org/jira/browse/HADOOP-14710
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Major
>
> Support data transfer between Hadoop and [AWS 
> Snowball|http://docs.aws.amazon.com/snowball/latest/ug/whatissnowball.html].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14060:

Target Version/s: 3.3.0  (was: 3.0.1)

> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine 
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and 
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that 
> /logs should subject to the same access control as intended by the original 
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in 
> {{HttpServer2}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595989#comment-16595989
 ] 

John Zhuge commented on HADOOP-14060:
-

No time for 3.2. Moved out.

> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine 
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and 
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that 
> /logs should subject to the same access control as intended by the original 
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in 
> {{HttpServer2}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14060:

Target Version/s: 3.0.1  (was: 3.0.1, 3.2.0)

> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine 
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and 
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that 
> /logs should subject to the same access control as intended by the original 
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in 
> {{HttpServer2}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594308#comment-16594308
 ] 

John Zhuge commented on HADOOP-15633:
-

Committed to branch-3.1 and branch-3.0.

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, 
> HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
>   

[jira] [Updated] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15633:

   Resolution: Fixed
Fix Version/s: 3.2.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Tranks [~ferhui] for discovering and fixing the issue!

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, 
> HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at 

[jira] [Assigned] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-12125:
---

Assignee: John Zhuge  (was: Rushabh S Shah)

> Retrying UnknownHostException on a proxy does not actually retry hostname 
> resolution
> 
>
> Key: HADOOP-12125
> URL: https://issues.apache.org/jira/browse/HADOOP-12125
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Reporter: Jason Lowe
>Assignee: John Zhuge
>Priority: Major
>
> When RetryInvocationHandler attempts to retry an UnknownHostException the 
> hostname fails to be resolved again.  The InetSocketAddress in the 
> ConnectionId has cached the fact that the hostname is unresolvable, and when 
> the proxy tries to setup a new Connection object with that ConnectionId it 
> checks if the (cached) resolution result is unresolved and immediately throws.
> The end result is we sleep and retry for no benefit.  The hostname resolution 
> is never attempted again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583097#comment-16583097
 ] 

John Zhuge commented on HADOOP-15633:
-

+1 Patch 005 LGTM

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, 
> HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> at 

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582039#comment-16582039
 ] 

John Zhuge commented on HADOOP-15633:
-

Great work [~ferhui]!

TrashPolicyDefault.java
* #158: Remove the space after “new Path ”
* #162: Adding “continue” is not quite right, because it takes us to the “for” 
loop that only loops twice, and for a different purpose. As a quick hack, you 
might add  a line “—I;” before “continue;”  Any better solution?


> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, 
> HADOOP-15633.003.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> 

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581903#comment-16581903
 ] 

John Zhuge commented on HADOOP-15633:
-

[~ferhui] Thanks for discovering the issue and creating a patch!

TrashPolicyDefault.java
* #152: The line exceeds 80 chars
* #154: Add space after “while”
* #158: Do not wrap a line before a comma
* #160: Do not mkdir here inside catch block, loop back to the try block 
instead, for 2 reasons: 1) there might be more conflicts, 2) mkdirs may throw 
exception.

TestTrash.java
* The new unit test seems to be inserted into the middle of another test case 
“Verify old checkpoint format is recognized”. 
* Can we create a new test case if possible? Test case “trashShell” is way too 
long!


> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> 

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579224#comment-16579224
 ] 

John Zhuge commented on HADOOP-15633:
-

Thanks for patch 002. I will review.

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at 

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559932#comment-16559932
 ] 

John Zhuge commented on HADOOP-15633:
-

After "hadoop fs -rm /user/hadoop/aaa/bbb/ccc", expect 
"/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc", since 
"/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb" is an existing file.

 

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at 

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559195#comment-16559195
 ] 

John Zhuge commented on HADOOP-15633:
-

[~ferhui] Sorry, what I meant was "name conflict resolution is not done for a 
parent dir on the path." for your command:
{noformat}
hadoop fs -rm /user/hadoop/aaa/bbb/ccc{noformat}

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at 
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at 
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at 

[jira] [Comment Edited] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558647#comment-16558647
 ] 

John Zhuge edited comment on HADOOP-15633 at 7/26/18 5:29 PM:
--

If you run the following command after your test steps:
{noformat}
hadoop fs -rm -r /user/hadoop/aaa/bbb
{noformat}
You will see the name conflict resolution, "bbb" -> "bbb1532625817927":
{noformat}
# hadoop fs -ls -R /user/hadoop/.Trash
drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current
drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user
drwx-- - hadoop hadoop 0 2018-07-26 17:21 
/user/hadoop/.Trash/Current/user/hadoop
drwx-- - hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa
-rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:21 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
drwxr-xr-x - hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927
-rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc{noformat}
Looks like name conflict resolution is not done for a parent dir on the path.

 


was (Author: jzhuge):
If you run the following command after your test steps:
{noformat}
hadoop fs -rm -r /user/hadoop/aaa/bbb
{noformat}
You will see the name conflict resolution, "bbb" -> "bbb1532625817927":

 
{noformat}
# hadoop fs -ls -R /user/hadoop/.Trash
drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current
drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user
drwx-- - hadoop hadoop 0 2018-07-26 17:21 
/user/hadoop/.Trash/Current/user/hadoop
drwx-- - hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa
-rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:21 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
drwxr-xr-x - hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927
-rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc{noformat}
Looks like name conflict resolution is not done for a parent dir on the path.

 

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at 

[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558647#comment-16558647
 ] 

John Zhuge commented on HADOOP-15633:
-

If you run the following command after your test steps:
{noformat}
hadoop fs -rm -r /user/hadoop/aaa/bbb
{noformat}
You will see the name conflict resolution, "bbb" -> "bbb1532625817927":

 
{noformat}
# hadoop fs -ls -R /user/hadoop/.Trash
drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current
drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user
drwx-- - hadoop hadoop 0 2018-07-26 17:21 
/user/hadoop/.Trash/Current/user/hadoop
drwx-- - hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa
-rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:21 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
drwxr-xr-x - hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927
-rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:23 
/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc{noformat}
Looks like name conflict resolution is not done for a parent dir on the path.

 

> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors 
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: 
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: 
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at 
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at 
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at 

[jira] [Commented] (HADOOP-14435) TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506606#comment-16506606
 ] 

John Zhuge commented on HADOOP-14435:
-

Saw it on 3.0.3 (RC0).

> TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed
> --
>
> Key: HADOOP-14435
> URL: https://issues.apache.org/jira/browse/HADOOP-14435
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation, fs/adl
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Saw the following assertion failure in branch-2 and trunk:
> {noformat}
> Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 80.189 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive
> testMkdirsWithUmask(org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive)
>   Time elapsed: 0.71 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<461> but was:<456>
>   at junit.framework.Assert.fail(Assert.java:57)
>   at junit.framework.Assert.failNotEquals(Assert.java:329)
>   at junit.framework.Assert.assertEquals(Assert.java:78)
>   at junit.framework.Assert.assertEquals(Assert.java:219)
>   at junit.framework.Assert.assertEquals(Assert.java:226)
>   at junit.framework.TestCase.assertEquals(TestCase.java:392)
>   at 
> org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsWithUmask(FileSystemContractBaseTest.java:242)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at 
> org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive.runTest(TestAdlFileSystemContractLive.java:59)
> Results :
> Failed tests:
>   
> TestAdlFileSystemContractLive.runTest:59->FileSystemContractBaseTest.testMkdirsWithUmask:242
>  expected:<461> but was:<456>
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14435) TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14435:

Issue Type: Bug  (was: Task)

> TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed
> --
>
> Key: HADOOP-14435
> URL: https://issues.apache.org/jira/browse/HADOOP-14435
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation, fs/adl
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Saw the following assertion failure in branch-2 and trunk:
> {noformat}
> Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 80.189 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive
> testMkdirsWithUmask(org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive)
>   Time elapsed: 0.71 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<461> but was:<456>
>   at junit.framework.Assert.fail(Assert.java:57)
>   at junit.framework.Assert.failNotEquals(Assert.java:329)
>   at junit.framework.Assert.assertEquals(Assert.java:78)
>   at junit.framework.Assert.assertEquals(Assert.java:219)
>   at junit.framework.Assert.assertEquals(Assert.java:226)
>   at junit.framework.TestCase.assertEquals(TestCase.java:392)
>   at 
> org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsWithUmask(FileSystemContractBaseTest.java:242)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at 
> org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive.runTest(TestAdlFileSystemContractLive.java:59)
> Results :
> Failed tests:
>   
> TestAdlFileSystemContractLive.runTest:59->FileSystemContractBaseTest.testMkdirsWithUmask:242
>  expected:<461> but was:<456>
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14175) NPE when ADL store URI contains underscore

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421110#comment-16421110
 ] 

John Zhuge commented on HADOOP-14175:
-

[~keuler] Thanks for the contribution. I have added you as a contributor to 
Hadoop Common project.

> NPE when ADL store URI contains underscore
> --
>
> Key: HADOOP-14175
> URL: https://issues.apache.org/jira/browse/HADOOP-14175
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: Xiaobing Zhou
>Priority: Minor
>  Labels: newbie, supportability
>
> Please note the underscore {{_}} in the store name {{jzhuge_adls}}. Same NPE 
> wherever the underscore in the URI.
> {noformat}
> $ bin/hadoop fs -ls adl://jzhuge_adls.azuredatalakestore.net/
> -ls: Fatal internal error
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.fs.adl.AdlFileSystem.initialize(AdlFileSystem.java:145)
>   at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3257)
>   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
>   at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3306)
>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3274)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
>   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
>   at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
>   at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:249)
>   at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:232)
>   at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
>   at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399707#comment-16399707
 ] 

John Zhuge commented on HADOOP-12125:
-

[~shahrs87] and [~jlowe], any progress? We hit the same issue when the non-HA 
NN went down and AWS spun up another NN instance with a different IP address. 
Both Job History Server and Spark History Server were stuck because 
NameNodeProxy held on to the old IP address.

> Retrying UnknownHostException on a proxy does not actually retry hostname 
> resolution
> 
>
> Key: HADOOP-12125
> URL: https://issues.apache.org/jira/browse/HADOOP-12125
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Reporter: Jason Lowe
>Assignee: Rushabh S Shah
>Priority: Major
>
> When RetryInvocationHandler attempts to retry an UnknownHostException the 
> hostname fails to be resolved again.  The InetSocketAddress in the 
> ConnectionId has cached the fact that the hostname is unresolvable, and when 
> the proxy tries to setup a new Connection object with that ConnectionId it 
> checks if the (cached) resolution result is unresolved and immediately throws.
> The end result is we sleep and retry for no benefit.  The hostname resolution 
> is never attempted again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12832) Implement unix-like 'FsShell -touch'

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388350#comment-16388350
 ] 

John Zhuge commented on HADOOP-12832:
-

Thanks [~bharatviswa] for taking this up.

> Implement unix-like 'FsShell -touch' 
> -
>
> Key: HADOOP-12832
> URL: https://issues.apache.org/jira/browse/HADOOP-12832
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: Gera Shegalov
>Assignee: Bharat Viswanadham
>Priority: Major
>
> We needed to touch a bunch of files as in 
> https://en.wikipedia.org/wiki/Touch_(Unix) . 
> Because FsShell does not expose FileSystem#setTimes  , we had to do it 
> programmatically in Scalding REPL. Seems like it should not be this 
> complicated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12832) Implement unix-like 'FsShell -touch'

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388282#comment-16388282
 ] 

John Zhuge commented on HADOOP-12832:
-

Unfortunately don't have time to work on it for a while.

> Implement unix-like 'FsShell -touch' 
> -
>
> Key: HADOOP-12832
> URL: https://issues.apache.org/jira/browse/HADOOP-12832
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: Gera Shegalov
>Priority: Major
>
> We needed to touch a bunch of files as in 
> https://en.wikipedia.org/wiki/Touch_(Unix) . 
> Because FsShell does not expose FileSystem#setTimes  , we had to do it 
> programmatically in Scalding REPL. Seems like it should not be this 
> complicated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-12832) Implement unix-like 'FsShell -touch'

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-12832:
---

Assignee: (was: John Zhuge)

> Implement unix-like 'FsShell -touch' 
> -
>
> Key: HADOOP-12832
> URL: https://issues.apache.org/jira/browse/HADOOP-12832
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: Gera Shegalov
>Priority: Major
>
> We needed to touch a bunch of files as in 
> https://en.wikipedia.org/wiki/Touch_(Unix) . 
> Because FsShell does not expose FileSystem#setTimes  , we had to do it 
> programmatically in Scalding REPL. Seems like it should not be this 
> complicated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-12832) Implement unix-like 'FsShell -touch'

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-12832:
---

Assignee: John Zhuge

> Implement unix-like 'FsShell -touch' 
> -
>
> Key: HADOOP-12832
> URL: https://issues.apache.org/jira/browse/HADOOP-12832
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: Gera Shegalov
>Assignee: John Zhuge
>Priority: Major
>
> We needed to touch a bunch of files as in 
> https://en.wikipedia.org/wiki/Touch_(Unix) . 
> Because FsShell does not expose FileSystem#setTimes  , we had to do it 
> programmatically in Scalding REPL. Seems like it should not be this 
> complicated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work stopped] (HADOOP-12832) Implement unix-like 'FsShell -touch'

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-12832 stopped by John Zhuge.
---
> Implement unix-like 'FsShell -touch' 
> -
>
> Key: HADOOP-12832
> URL: https://issues.apache.org/jira/browse/HADOOP-12832
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: Gera Shegalov
>Assignee: John Zhuge
>Priority: Major
>
> We needed to touch a bunch of files as in 
> https://en.wikipedia.org/wiki/Touch_(Unix) . 
> Because FsShell does not expose FileSystem#setTimes  , we had to do it 
> programmatically in Scalding REPL. Seems like it should not be this 
> complicated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-12832) Implement unix-like 'FsShell -touch'

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-12832:
---

Assignee: (was: John Zhuge)

> Implement unix-like 'FsShell -touch' 
> -
>
> Key: HADOOP-12832
> URL: https://issues.apache.org/jira/browse/HADOOP-12832
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: Gera Shegalov
>Priority: Major
>
> We needed to touch a bunch of files as in 
> https://en.wikipedia.org/wiki/Touch_(Unix) . 
> Because FsShell does not expose FileSystem#setTimes  , we had to do it 
> programmatically in Scalding REPL. Seems like it should not be this 
> complicated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366130#comment-16366130
 ] 

John Zhuge commented on HADOOP-13972:
-

Thanks [~ste...@apache.org] !

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366129#comment-16366129
 ] 

John Zhuge commented on HADOOP-14060:
-

Sorry, don't have time to work on it now. Unassigned the JIRA.

> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine 
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and 
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that 
> /logs should subject to the same access control as intended by the original 
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in 
> {{HttpServer2}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-14060:
---

Assignee: (was: John Zhuge)

> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine 
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and 
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that 
> /logs should subject to the same access control as intended by the original 
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in 
> {{HttpServer2}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363513#comment-16363513
 ] 

John Zhuge commented on HADOOP-13972:
-

Thanks [~ssonker] for the PR. Added you as a contributor to Hadoop Common and 
assigned the Jira to you.

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-13972) ADLS to support per-store configuration

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-13972:
---

Assignee: Sharad Sonker

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge resolved HADOOP-14961.
-
Resolution: Duplicate

Fixed by HADOOP-14816.

> Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
> --
>
> Key: HADOOP-14961
> URL: https://issues.apache.org/jira/browse/HADOOP-14961
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build, test
>Affects Versions: 3.1.0
>Reporter: John Zhuge
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console 
> {noformat} 
> Downloading Oracle Java 8... 
> --2017-10-18 18:28:11-- 
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>  
> Resolving download.oracle.com (download.oracle.com)... 
> 23.59.190.131, 23.59.190.130 
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... 
> connected. 
> HTTP request sent, awaiting response... 302 Moved Temporarily 
> Location: 
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>  [following] 
> --2017-10-18 18:28:11-- 
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>  
> Resolving edelivery.oracle.com (edelivery.oracle.com)... 
> 23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e 
> Connecting to edelivery.oracle.com 
> (edelivery.oracle.com)|23.39.16.136|:443... connected. 
> HTTP request sent, awaiting response... 302 Moved 
> Temporarily 
> Location: 
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>  [following] 
> --2017-10-18 18:28:11-- 
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>  
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... 
> connected. 
> HTTP request sent, awaiting response... 404 Not Found 
> 2017-10-18 18:28:12 ERROR 404: Not Found. 
> download failed 
> Oracle JDK 8 is NOT installed. 
> {noformat}
> Looks like Oracle JDK 8u144 is no longer available for download using that 
> link. 8u151 and 8u152 are available.
> Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs 
> failed the same way, all on build host H1 and H6.
> [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial" 
> for a long term fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reopened HADOOP-14961:
-

> Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
> --
>
> Key: HADOOP-14961
> URL: https://issues.apache.org/jira/browse/HADOOP-14961
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build, test
>Affects Versions: 3.1.0
>Reporter: John Zhuge
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console 
> {noformat} 
> Downloading Oracle Java 8... 
> --2017-10-18 18:28:11-- 
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>  
> Resolving download.oracle.com (download.oracle.com)... 
> 23.59.190.131, 23.59.190.130 
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... 
> connected. 
> HTTP request sent, awaiting response... 302 Moved Temporarily 
> Location: 
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>  [following] 
> --2017-10-18 18:28:11-- 
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>  
> Resolving edelivery.oracle.com (edelivery.oracle.com)... 
> 23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e 
> Connecting to edelivery.oracle.com 
> (edelivery.oracle.com)|23.39.16.136|:443... connected. 
> HTTP request sent, awaiting response... 302 Moved 
> Temporarily 
> Location: 
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>  [following] 
> --2017-10-18 18:28:11-- 
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>  
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... 
> connected. 
> HTTP request sent, awaiting response... 404 Not Found 
> 2017-10-18 18:28:12 ERROR 404: Not Found. 
> download failed 
> Oracle JDK 8 is NOT installed. 
> {noformat}
> Looks like Oracle JDK 8u144 is no longer available for download using that 
> link. 8u151 and 8u152 are available.
> Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs 
> failed the same way, all on build host H1 and H6.
> [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial" 
> for a long term fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14651) Update okhttp version to 2.7.5

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343006#comment-16343006
 ] 

John Zhuge commented on HADOOP-14651:
-

Fine with me.

> Update okhttp version to 2.7.5
> --
>
> Key: HADOOP-14651
> URL: https://issues.apache.org/jira/browse/HADOOP-14651
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-14651.001.patch, HADOOP-14651.002.patch, 
> HADOOP-14651.003.patch, HADOOP-14651.004.patch
>
>
> The current artifact is:
> com.squareup.okhttp:okhttp:2.4.0
> That version could either be bumped to 2.7.5 (the latest of that line), or 
> use the latest artifact:
> com.squareup.okhttp3:okhttp:3.8.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15134) ADL problems parsing JSON responses to include error details

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15134:

Labels: supportability  (was: )

> ADL problems parsing JSON responses to include error details
> 
>
> Key: HADOOP-15134
> URL: https://issues.apache.org/jira/browse/HADOOP-15134
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>  Labels: supportability
>
> Currently any failure of ADL's response JSON parsing results in the error 
> text like "Unexpected error happened reading response stream or parsing JSon 
> from rename()";". This is not useful. Fix by: including the exception text, 
> logging the Ex & info



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16282508#comment-16282508
 ] 

John Zhuge commented on HADOOP-15056:
-

+1 LGTM

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, 
> HADOOP-15056.003.patch, HADOOP-15056.004.patch, HADOOP-15056.005.patch, 
> HADOOP-15056.006.patch, HADOOP-15056.007.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278301#comment-16278301
 ] 

John Zhuge commented on HADOOP-15056:
-

Here is a table of all scenarios and the comparisons of patches and original 
behavior. Logging and exception only occurs when explicitly indicated.
|| instanceof StreamCapabilities || has UNBUFFER capaility || instanceof 
CanUnbuffer || Patch 004 behavior || Patch 002 behavior || Original behavior ||
| Yes | Yes | Yes | Call unbuffer | Call unbuffer | Call unbuffer |
| Yes | Yes | No | UOE | UOE | UOE |
| Yes | No | N/A | LOG | No-op | No-op |
| No | N/A | Yes | LOG | Call unbuffer |  No-op |
| No | N/A | No | No-op | UOE |  No-op |


> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, 
> HADOOP-15056.003.patch, HADOOP-15056.004.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14294) Rename ADLS mountpoint properties

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276069#comment-16276069
 ] 

John Zhuge commented on HADOOP-14294:
-

Excellent! I have added you as a Hadoop Common contributor and assigned this 
JIRA to you.

> Rename ADLS mountpoint properties
> -
>
> Key: HADOOP-14294
> URL: https://issues.apache.org/jira/browse/HADOOP-14294
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: Rohan Garg
>
> Follow up to HADOOP-14038. Rename the prefix of 
> {{dfs.adls..mountpoint}} and {{dfs.adls..hostname}} to 
> {{fs.adl.}}.
> Borrow code from 
> https://issues.apache.org/jira/secure/attachment/12857500/HADOOP-14038.006.patch
>  and add a few unit tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-14294) Rename ADLS mountpoint properties

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-14294:
---

Assignee: Rohan Garg  (was: John Zhuge)

> Rename ADLS mountpoint properties
> -
>
> Key: HADOOP-14294
> URL: https://issues.apache.org/jira/browse/HADOOP-14294
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: Rohan Garg
>
> Follow up to HADOOP-14038. Rename the prefix of 
> {{dfs.adls..mountpoint}} and {{dfs.adls..hostname}} to 
> {{fs.adl.}}.
> Borrow code from 
> https://issues.apache.org/jira/secure/attachment/12857500/HADOOP-14038.006.patch
>  and add a few unit tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272192#comment-16272192
 ] 

John Zhuge commented on HADOOP-15056:
-

IMHO, it is not a good idea to swallow CCE and just log.

In the try block, StreamCapabilities#hasCapabilities has to be called, because 
this is the new algorithm introduced by HADOOP-15012.: an input stream should 
be queried about its capability UNBUFFER before unbuffer is called. Casting to 
CanUnbuffer should always succeed in this case.

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272100#comment-16272100
 ] 

John Zhuge commented on HADOOP-15056:
-

No worry. Good discussion.

IMPALA-5909 complained excessive UOE exceptions.

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271601#comment-16271601
 ] 

John Zhuge commented on HADOOP-15056:
-

StreamCapabilities was added by HDFS-11644. It is considered a more flexible 
approach than {{instanceof}}.

bq. it may prevent helpful debugging information to be passed back to the user

Could you elaborate?

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15056:

Attachment: HADOOP-15056.002.patch

Hi [~jackbearden], I think StreamCapabilitiesPolicy#unbuffer should still call 
hasCapability if the input stream implements interface StreamCapabilities. 
Uploaded patch 002. Let me know what you think.

Testing Done
* TestCryptoStreams
* TestCryptoStreamsForLocalFS
* TestCryptoStreamsNormal
* TestHdfsCryptoStreams
* TestUnbuffer

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15056:

Summary: Fix TestUnbuffer#testUnbufferException failure  (was: Fix for 
TestUnbuffer#testUnbufferException Test)

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix for TestUnbuffer#testUnbufferException Test

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269484#comment-16269484
 ] 

John Zhuge commented on HADOOP-15056:
-

Just back from vacation, sorry for the delay. I will review shortly.

> Fix for TestUnbuffer#testUnbufferException Test
> ---
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix for TestUnbuffer#testUnbufferException Test

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260286#comment-16260286
 ] 

John Zhuge commented on HADOOP-15056:
-

[~jackbearden] Welcome to the Hadoop Open Source community! Your contribution 
is very much appreciated!

I have added you as a contributor and assigned the JIRA to you. Please take a 
look at https://wiki.apache.org/hadoop/HowToContribute. The section "Naming 
your patch" recommends the version number to be embedded in the patch file name.

I will take a look at your patch shortly.

> Fix for TestUnbuffer#testUnbufferException Test
> ---
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-15056) Fix for TestUnbuffer#testUnbufferException Test

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-15056:
---

Assignee: Jack Bearden

> Fix for TestUnbuffer#testUnbufferException Test
> ---
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge reassigned HADOOP-15012:
---

Assignee: John Zhuge

> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15012:

Target Version/s: 3.1.0  (was: 2.10.0)

> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge resolved HADOOP-15012.
-
   Resolution: Fixed
Fix Version/s: 3.1.0

Committed to trunk together with HADOOP-14872. Code review was done there.
{noformat}
6c32ddad302 HADOOP-14872. CryptoInputStream should implement unbuffer. 
Contributed by John Zhuge.
bf6a660232b HADOOP-15012. Add readahead, dropbehind, and unbuffer to 
StreamCapabilities. Contributed by John Zhuge.
{noformat}


> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

   Resolution: Fixed
Fix Version/s: 3.1.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks [~xiaochen] and [~steve_l] for the great reviews!

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246199#comment-16246199
 ] 

John Zhuge commented on HADOOP-14872:
-

Ran "test-patch" locally and got all +1s except these expected javac 
deprecation warnings. Committing.
{noformat}
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[42,46]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[251,30]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[252,33]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[32,46]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[40,46]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[201,16]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[204,16]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[362,25]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
[WARNING] 
/home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[364,25]
 [deprecation] StreamCapability in StreamCapabilities has been deprecated
{noformat}


> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Open  (was: Patch Available)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Patch Available  (was: Open)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Open  (was: Patch Available)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Patch Available  (was: Open)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: In Progress  (was: Patch Available)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Patch Available  (was: In Progress)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: In Progress  (was: Patch Available)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Patch Available  (was: In Progress)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Component/s: security

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Open  (was: Patch Available)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Patch Available  (was: Open)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Patch Available  (was: Open)

Tried to trigger pre-commit.

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: Open  (was: Patch Available)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Target Version/s: 3.1.0
  Status: Patch Available  (was: In Progress)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Status: In Progress  (was: Patch Available)

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15012:

Summary: Add readahead, dropbehind, and unbuffer to StreamCapabilities  
(was: Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities)

> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15012) Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-15012:

Summary: Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities  
(was: Enhance StreamCapabilities with READAHEAD, DROPBEHIND, and UNBUFFER)

> Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Priority: Major
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-14872:

Attachment: HADOOP-14872.013.patch

Thanks Xiao and Steve for the patience and great discussions.

Patch 013
* Incorporate Xiao and Steve's comments
* Add {{StreamCapabilitiesPolicy#unbuffer}} to implement the policy for 
unbuffer. Java 8 static method for interface CanUnbuffer could be a good choice 
but we can't use it due to backport concern.
* Use a table in FileSystem.md to list capabilities
* Plan to commit StreamCapabilities enhancements in HADOOP-15012 first, then 
check in CryptoInputStream changes here.

> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Major
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, 
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, 
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, 
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, 
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
>   CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, 
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
>   ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
>   throw new UnsupportedOperationException("this stream does not " +
>   "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If 
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
>   DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very 
> well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15012) Enhance StreamCapabilities with READAHEAD, DROPBEHIND, and UNBUFFER

[John Zhuge (JIRA)]

John Zhuge created HADOOP-15012:
---

 Summary: Enhance StreamCapabilities with READAHEAD, DROPBEHIND, 
and UNBUFFER
 Key: HADOOP-15012
 URL: https://issues.apache.org/jira/browse/HADOOP-15012
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.9.0
Reporter: John Zhuge
Priority: Major


A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
class with READAHEAD, DROPBEHIND, and UNBUFFER capability.

Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org