[jira] [Commented] (HADOOP-12719) Implement lease recover in wasb file system

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026817#comment-16026817
 ] 

Chuan Liu commented on HADOOP-12719:


I am not sure lease recovery applies to WASB since it is a HDFS internal 
implementation. There may be some contracts defined on the fs API that need to 
be met, e.g. hflush()/hsync() need to guarantee data is persisted. I haven't 
worked on this for some time. HBase on Azure is a supported feature for some 
time. I think you can contact the production team in Azure to check if they 
have already supported your scenario.

> Implement lease recover in wasb file system
> ---
>
> Key: HADOOP-12719
> URL: https://issues.apache.org/jira/browse/HADOOP-12719
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: Liu Shaohui
>
> HBase depend on the lease recovery of file system to prevent the write 
> operations from the "dead" regionservers. 
> After HADOOP-9629, hadoop supports Azure Storage as an alternative Hadoop 
> Compatible File System. If we want to deploy HBase on Azure storage and keep 
> data safety, we need to implement lease recover in wasb file system.
> [~cnauroth] [~chuanliu]
> I don't know much about wasb. Any suggestions about how to implements this? 
> Eg, changing permission of the hlog to readonly like FSMapRUtils does?
> Thanks~



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13223) winutils.exe is a bug nexus and should be killed with an axe.

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310835#comment-15310835
 ] 

Chuan Liu commented on HADOOP-13223:


I can add some context here. Back then:
# Native library inclusion is *optional* in Hadoop on both Windows and Linux.
# Accessing Linux cluster from Windows client is not supported.
# There is no ASF build of Hadoop on Windows (due to Apache had no Windows CI 
machine).

All native and Java API gaps are addressed in calling to external commands due 
to 1). The original Hadoop on Windows promise is that Windows implementation 
should never break Linux side. We agreed to create "winutils" to address 
missing command line utilities on Windows when porting Hadoop to Windows. 
Later, when fixing some other file IO issues, we make the native library a 
mandate on Windows, but the existing "winutils.exe" is not replaced with JNI 
calls due to the amount of engineering work involved.

I read through your complaints, I think problem 1 & 2 are a distro issue and if 
Apache has an official build that include Windows binaries that can help with 
the problem. [~ste...@apache.org] also provides a partial solution. Problem 3 - 
6 can be summarized as poor error messages when calling external commands on 
Windows. So I agree with that and there are various places such error message 
should be improved. However, I do not think removing "winutils.exe" is a fix to 
all your problem. It will likely just replace ".exe" problem with ".dll" 
problem.

That said, personally, I am also in favor to get rid of the "winutils.exe" and 
replace necessary calls with the JNI implementation. "winutils" code is 
designed to have all main implementations in "libwintuils", so both command 
line implemetation "winutils.exe" and JNI implementation "hadoop.dll" are 
surface level wrappers that statically link to the same underlying library. On 
this front, it should not be too difficult to move all "winutils.exe" 
implemetations into JNI calls.

> winutils.exe is a bug nexus and should be killed with an axe.
> -
>
> Key: HADOOP-13223
> URL: https://issues.apache.org/jira/browse/HADOOP-13223
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin
>Affects Versions: 2.6.0
> Environment: Microsoft Windows, all versions
>Reporter: john lilley
>
> winutils.exe was apparently created as a stopgap measure to allow Hadoop to 
> "work" on Windows platforms, because the NativeIO libraries aren't 
> implemented there (edit: even NativeIO probably doesn't cover the operations 
> that winutils.exe is used for).  Rather than building a DLL that makes native 
> OS calls, the creators of winutils.exe must have decided that it would be 
> more expedient to create an EXE to carry out file system operations in a 
> linux-like fashion.  Unfortunately, like many stopgap measures in software, 
> this one has persisted well beyond its expected lifetime and usefulness.  My 
> team creates software that runs on Windows and Linux, and winutils.exe is 
> probably responsible for 20% of all issues we encounter, both during 
> development and in the field.
> Problem #1 with winutils.exe is that it is simply missing from many popular 
> distros and/or the client-side software installation for said distros, when 
> supplied, fails to install winutils.exe.  Thus, as software developers, we 
> are forced to pick one version and distribute and install it with our 
> software.
> Which leads to problem #2: winutils.exe are not always compatible.  In 
> particular, MapR MUST have its winutils.exe in the system path, but doing so 
> breaks the Hadoop distro for every other Hadoop vendor.  This makes creating 
> and maintaining test environments that work with all of the Hadoop distros we 
> want to test unnecessarily tedious and error-prone.
> Problem #3 is that the mechanism by which you inform the Hadoop client 
> software where to find winutils.exe is poorly documented and fragile.  First, 
> it can be in the PATH.  If it is in the PATH, that is where it is found.  
> However, the documentation, such as it is, makes no mention of this, and 
> instead says that you should set the HADOOP_HOME environment variable, which 
> does NOT override the winutils.exe found in your system PATH.
> Which leads to problem #4: There is no logging that says where winutils.exe 
> was actually found and loaded.  Because of this, fixing problems of finding 
> the wrong winutils.exe are extremely difficult.
> Problem #5 is that most of the time, such as when accessing straight up HDFS 
> and YARN, one does not *need* winutils.exe.  But if it is missing, the log 
> messages complain about its absence.  When we are trying to diagnose an 
> obscure issue in Hadoop (of which there are many), the presence of this red 

[jira] [Commented] (HADOOP-11691) X86 build of libwinutils is broken

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382665#comment-14382665
 ] 

Chuan Liu commented on HADOOP-11691:


+1 I have verified the CPU feature is present when building with the latest 
patch. Thanks for fixing the build! 

 X86 build of libwinutils is broken
 --

 Key: HADOOP-11691
 URL: https://issues.apache.org/jira/browse/HADOOP-11691
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, native
Affects Versions: 2.7.0
Reporter: Remus Rusanu
Assignee: Kiran Kumar M R
Priority: Critical
 Attachments: HADOOP-11691-001.patch, HADOOP-11691-002.patch, 
 HADOOP-11691-003.patch


 Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results 
 in error:
 {code}
 (Link target) -
   
 E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
  : fatal error LNK1112: module machine type 'x64' conflicts with target 
 machine type 'X86' 
 [E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11691) X86 build of libwinutils is broken

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370382#comment-14370382
 ] 

Chuan Liu commented on HADOOP-11691:


[~kiranmr], thanks for the patch! It looks good, but when I build the winutils 
binary with the patch, the CPU rate control feature does not present in the 
binary. I suspect it is related to the {{$(VCInstallDir)}} paths missed in the 
patch.

 X86 build of libwinutils is broken
 --

 Key: HADOOP-11691
 URL: https://issues.apache.org/jira/browse/HADOOP-11691
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, native
Affects Versions: 3.0.0
Reporter: Remus Rusanu
Assignee: Kiran Kumar M R
 Attachments: HADOOP-11691-001.patch, HADOOP-11691-002.patch


 Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results 
 in error:
 {code}
 (Link target) -
   
 E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
  : fatal error LNK1112: module machine type 'x64' conflicts with target 
 machine type 'X86' 
 [E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11691) X86 build of libwinutils is broken

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353293#comment-14353293
 ] 

Chuan Liu commented on HADOOP-11691:


Sorry for the build break. I did not test the YARN-2190 on x86 platform. 
[~kiranmr], do you want to work on a fix? Otherwise, I will try to port the 
change to Win32 later this week. I think WinSDK8 is already conditioned on the 
Windows SDK 8.1. If there is no Windows 8.1 SDK installed, there should be no 
build break. 

 X86 build of libwinutils is broken
 --

 Key: HADOOP-11691
 URL: https://issues.apache.org/jira/browse/HADOOP-11691
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, native
Affects Versions: 3.0.0
Reporter: Remus Rusanu
Assignee: Kiran Kumar M R

 Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results 
 in error:
 {code}
 (Link target) -
   
 E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
  : fatal error LNK1112: module machine type 'x64' conflicts with target 
 machine type 'X86' 
 [E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11163) MetricsSystemImpl may miss a registered source

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-11163:
--

 Summary: MetricsSystemImpl may miss a registered source
 Key: HADOOP-11163
 URL: https://issues.apache.org/jira/browse/HADOOP-11163
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor


Calling to {{MetricsSystemImpl#register(String name, String desc, T source)}} 
could miss when the name is {{null}}. This is related to my previous change in 
HADOOP-11105.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11163) MetricsSystemImpl may miss a registered source

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-11163:
---
Status: Patch Available  (was: Open)

 MetricsSystemImpl may miss a registered source
 --

 Key: HADOOP-11163
 URL: https://issues.apache.org/jira/browse/HADOOP-11163
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-11163.patch


 Calling to {{MetricsSystemImpl#register(String name, String desc, T source)}} 
 could miss when the name is {{null}}. This is related to my previous change 
 in HADOOP-11105.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11163) MetricsSystemImpl may miss a registered source

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-11163:
---
Attachment: HADOOP-11163.patch

Attach a patch.

 MetricsSystemImpl may miss a registered source
 --

 Key: HADOOP-11163
 URL: https://issues.apache.org/jira/browse/HADOOP-11163
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-11163.patch


 Calling to {{MetricsSystemImpl#register(String name, String desc, T source)}} 
 could miss when the name is {{null}}. This is related to my previous change 
 in HADOOP-11105.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11105) MetricsSystemImpl could leak memory in registered callbacks

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-11105:
--

 Summary: MetricsSystemImpl could leak memory in registered 
callbacks
 Key: HADOOP-11105
 URL: https://issues.apache.org/jira/browse/HADOOP-11105
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu


In {{MetricsSystemImpl.register(final String name, final String description, 
final T sink)}} method, we will also add a callback function to the 
{{callbacks}} array list. However, upon unregistering, the callback function is 
not removed from the list in existing code. As a result, the memory allocated 
for the callback function could accumulate in the {{MetricsSystemImpl}} list. 
In one of our cluster, we have seen a case where there are about 874 MB memory 
retained by {{MetricsSystemImpl}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11105) MetricsSystemImpl could leak memory in registered callbacks

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-11105:
---
Attachment: memory_leak_2.png
memory_leak_1.png

Attaching two eclipse Memory Analyzer graph of the heap dump.

 MetricsSystemImpl could leak memory in registered callbacks
 ---

 Key: HADOOP-11105
 URL: https://issues.apache.org/jira/browse/HADOOP-11105
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: memory_leak_1.png, memory_leak_2.png


 In {{MetricsSystemImpl.register(final String name, final String description, 
 final T sink)}} method, we will also add a callback function to the 
 {{callbacks}} array list. However, upon unregistering, the callback function 
 is not removed from the list in existing code. As a result, the memory 
 allocated for the callback function could accumulate in the 
 {{MetricsSystemImpl}} list. In one of our cluster, we have seen a case where 
 there are about 874 MB memory retained by {{MetricsSystemImpl}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11105) MetricsSystemImpl could leak memory in registered callbacks

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-11105:
---
Attachment: HADOOP-11105.patch

Attaching a patch.

 MetricsSystemImpl could leak memory in registered callbacks
 ---

 Key: HADOOP-11105
 URL: https://issues.apache.org/jira/browse/HADOOP-11105
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-11105.patch, memory_leak_1.png, memory_leak_2.png


 In {{MetricsSystemImpl.register(final String name, final String description, 
 final T sink)}} method, we will also add a callback function to the 
 {{callbacks}} array list. However, upon unregistering, the callback function 
 is not removed from the list in existing code. As a result, the memory 
 allocated for the callback function could accumulate in the 
 {{MetricsSystemImpl}} list. In one of our cluster, we have seen a case where 
 there are about 874 MB memory retained by {{MetricsSystemImpl}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11105) MetricsSystemImpl could leak memory in registered callbacks

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-11105:
---
Status: Patch Available  (was: Open)

 MetricsSystemImpl could leak memory in registered callbacks
 ---

 Key: HADOOP-11105
 URL: https://issues.apache.org/jira/browse/HADOOP-11105
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-11105.patch, memory_leak_1.png, memory_leak_2.png


 In {{MetricsSystemImpl.register(final String name, final String description, 
 final T sink)}} method, we will also add a callback function to the 
 {{callbacks}} array list. However, upon unregistering, the callback function 
 is not removed from the list in existing code. As a result, the memory 
 allocated for the callback function could accumulate in the 
 {{MetricsSystemImpl}} list. In one of our cluster, we have seen a case where 
 there are about 874 MB memory retained by {{MetricsSystemImpl}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11105) MetricsSystemImpl could leak memory in registered callbacks

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-11105:
---
Attachment: HADOOP-11105.2.patch

Fixing the format.

 MetricsSystemImpl could leak memory in registered callbacks
 ---

 Key: HADOOP-11105
 URL: https://issues.apache.org/jira/browse/HADOOP-11105
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-11105.2.patch, HADOOP-11105.patch, 
 memory_leak_1.png, memory_leak_2.png


 In {{MetricsSystemImpl.register(final String name, final String description, 
 final T sink)}} method, we will also add a callback function to the 
 {{callbacks}} array list. However, upon unregistering, the callback function 
 is not removed from the list in existing code. As a result, the memory 
 allocated for the callback function could accumulate in the 
 {{MetricsSystemImpl}} list. In one of our cluster, we have seen a case where 
 there are about 874 MB memory retained by {{MetricsSystemImpl}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11080) Convert Windows native build in hadoop-common to use CMake.

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-11080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127881#comment-14127881
 ] 

Chuan Liu commented on HADOOP-11080:


I was able to build on Windows using CMake. I have no objection on the plan to 
convert to CMake.

Just add a note: I got some errors building Hadoop with CMake on Powershell. I 
did not spend time root causing the problem, and switched to use cmd instead. 
so I cannot tell it is a CMake/Maven/Powershell problem. We can investigate the 
scenario later if other people also build and fail on Powershell.

 Convert Windows native build in hadoop-common to use CMake.
 ---

 Key: HADOOP-11080
 URL: https://issues.apache.org/jira/browse/HADOOP-11080
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, native
Reporter: Chris Nauroth

 The Windows native build in hadoop-common currently relies on a set of 
 checked-in MSBuild project files.  The logic of these project files is 
 largely redundant with the logic of CMakeLists.txt.  It causes some dual 
 maintenance overhead, because certain build changes require separate changes 
 for both CMake and MSBuild.  This issue proposes to convert the Windows build 
 process to use CMake instead of checked-in project files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-10272) Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces in the path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10272:
---

Attachment: HADOOP-10272.patch

The root cause is that {{PathData.getStringForChildPath()}} will return a path 
string that is encoded, i.e. ' ' will be encoded as '%20'. This path string 
will later be used again in the Path constructor to pass to URI constructor, 
and lead to double encoding. The wrongly encoded path will later lead to a copy 
failure, because the wrongly encoded path is on longer the original user input 
path. I suspect Unix/Linux also has this problem unless the Java URI 
implementation is different on Unix/Linux platform. Attaching a patch that 
address the issue. 

 Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces 
 in the path
 ---

 Key: HADOOP-10272
 URL: https://issues.apache.org/jira/browse/HADOOP-10272
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.2.0
Reporter: Shuaishuai Nie
Assignee: Chuan Liu
 Attachments: HADOOP-10272.patch


 Repro steps:
 with folder structure like: /ab/c d/ef.txt
 hadoop command (hadoop fs -copyFromLocal /ab/ /) or (hadoop fs -copyFromLocal 
 /ab/c d/ /) fail with error:
 copyFromLocal: File file:/ab/c%20d/ef.txt does not exist
 However command (hadoop fs -copyFromLocal /ab/c d/ef.txt /) success.
 Seems like hadoop treat file and directory differently when copyFromLocal.
 This only happens in Hadoop 2 and causing 2 Hive unit test failures 
 (external_table_with_space_in_location_path.q and 
 load_hdfs_file_with_space_in_the_name.q).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HADOOP-10272) Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces in the path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10272:
---

Affects Version/s: 3.0.0
   Status: Patch Available  (was: Open)

 Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces 
 in the path
 ---

 Key: HADOOP-10272
 URL: https://issues.apache.org/jira/browse/HADOOP-10272
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.2.0, 3.0.0
Reporter: Shuaishuai Nie
Assignee: Chuan Liu
 Attachments: HADOOP-10272.patch


 Repro steps:
 with folder structure like: /ab/c d/ef.txt
 hadoop command (hadoop fs -copyFromLocal /ab/ /) or (hadoop fs -copyFromLocal 
 /ab/c d/ /) fail with error:
 copyFromLocal: File file:/ab/c%20d/ef.txt does not exist
 However command (hadoop fs -copyFromLocal /ab/c d/ef.txt /) success.
 Seems like hadoop treat file and directory differently when copyFromLocal.
 This only happens in Hadoop 2 and causing 2 Hive unit test failures 
 (external_table_with_space_in_location_path.q and 
 load_hdfs_file_with_space_in_the_name.q).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HADOOP-10272) Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces in the path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10272:
---

Resolution: Duplicate
Status: Resolved  (was: Patch Available)

OK. I just found out this is a duplicate of HDFS-4329 when trying to rebase my 
patch to trunk. Resolving as duplicated.

 Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces 
 in the path
 ---

 Key: HADOOP-10272
 URL: https://issues.apache.org/jira/browse/HADOOP-10272
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.2.0
Reporter: Shuaishuai Nie
Assignee: Chuan Liu
 Attachments: HADOOP-10272.patch


 Repro steps:
 with folder structure like: /ab/c d/ef.txt
 hadoop command (hadoop fs -copyFromLocal /ab/ /) or (hadoop fs -copyFromLocal 
 /ab/c d/ /) fail with error:
 copyFromLocal: File file:/ab/c%20d/ef.txt does not exist
 However command (hadoop fs -copyFromLocal /ab/c d/ef.txt /) success.
 Seems like hadoop treat file and directory differently when copyFromLocal.
 This only happens in Hadoop 2 and causing 2 Hive unit test failures 
 (external_table_with_space_in_location_path.q and 
 load_hdfs_file_with_space_in_the_name.q).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (HADOOP-10272) Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces in the path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu reassigned HADOOP-10272:
--

Assignee: Chuan Liu

 Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces 
 in the path
 ---

 Key: HADOOP-10272
 URL: https://issues.apache.org/jira/browse/HADOOP-10272
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.2.0
Reporter: Shuaishuai Nie
Assignee: Chuan Liu

 Repro steps:
 with folder structure like: /ab/c d/ef.txt
 hadoop command (hadoop fs -copyFromLocal /ab/ /) or (hadoop fs -copyFromLocal 
 /ab/c d/ /) fail with error:
 copyFromLocal: File file:/ab/c%20d/ef.txt does not exist
 However command (hadoop fs -copyFromLocal /ab/c d/ef.txt /) success.
 Seems like hadoop treat file and directory differently when copyFromLocal.
 This only happens in Hadoop 2 and causing 2 Hive unit test failures 
 (external_table_with_space_in_location_path.q and 
 load_hdfs_file_with_space_in_the_name.q).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HADOOP-10272) Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces in the path

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13881351#comment-13881351
 ] 

Chuan Liu commented on HADOOP-10272:


I can take a look at the root cause.

 Hadoop 2 -copyFromLocal fail when source is a folder and there are spaces 
 in the path
 ---

 Key: HADOOP-10272
 URL: https://issues.apache.org/jira/browse/HADOOP-10272
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.2.0
Reporter: Shuaishuai Nie
Assignee: Chuan Liu

 Repro steps:
 with folder structure like: /ab/c d/ef.txt
 hadoop command (hadoop fs -copyFromLocal /ab/ /) or (hadoop fs -copyFromLocal 
 /ab/c d/ /) fail with error:
 copyFromLocal: File file:/ab/c%20d/ef.txt does not exist
 However command (hadoop fs -copyFromLocal /ab/c d/ef.txt /) success.
 Seems like hadoop treat file and directory differently when copyFromLocal.
 This only happens in Hadoop 2 and causing 2 Hive unit test failures 
 (external_table_with_space_in_location_path.q and 
 load_hdfs_file_with_space_in_the_name.q).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9629:
--

Description: 
h2. Description
This JIRA incorporates adding a new file system implementation for accessing 
Windows Azure Storage - Blob from within Hadoop, such as using blobs as input 
to MR jobs or configuring MR jobs to put their output directly into blob 
storage.

h2. High level design
At a high level, the code here extends the FileSystem class to provide an 
implementation for accessing blob storage; the scheme wasb is used for 
accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI 
scheme: {code}wasb[s]://container@account/path/to/file{code} to address 
individual blobs. We use the standard Azure Java SDK 
(com.microsoft.windowsazure) to do most of the work. In order to map a 
hierarchical file system over the flat name-value pair nature of blob storage, 
we create a specially tagged blob named path/to/dir whenever we create a 
directory called path/to/dir, then files under that are stored as normal blobs 
path/to/dir/file. We have many metrics implemented for it using the Metrics2 
interface. Tests are implemented mostly using a mock implementation for the 
Azure SDK functionality, with an option to test against a real blob storage if 
configured (instructions provided inside in README.txt).

h2. Credits and history
This has been ongoing work for a while, and the early version of this work can 
be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll 
post the patch here for Hadoop trunk first, then post a patch for branch-1 as 
well for backporting the functionality if accepted. Credit for this work goes 
to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as 
well as multiple people who have taken over this work since then (hope I don't 
forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, 
[~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu].

h2. Test
Besides unit tests, we have used WASB as the default file system in our service 
product. (HDFS is also used but not as default file system.) Various different 
customer and test workloads have been run against clusters with such 
configurations for quite some time. The current version reflects to the version 
of the code tested and used in our production environment.

  was:
h2. Description
This JIRA incorporates adding a new file system implementation for accessing 
Windows Azure Storage - Blob from within Hadoop, such as using blobs as input 
to MR jobs or configuring MR jobs to put their output directly into blob 
storage.

h2. High level design
At a high level, the code here extends the FileSystem class to provide an 
implementation for accessing blob storage; the scheme wasb is used for 
accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI 
scheme: {code}wasb[s]://container@account/path/to/file{code} to address 
individual blobs. We use the standard Azure Java SDK 
(com.microsoft.windowsazure) to do most of the work. In order to map a 
hierarchical file system over the flat name-value pair nature of blob storage, 
we create a specially tagged blob named path/to/dir whenever we create a 
directory called path/to/dir, then files under that are stored as normal blobs 
path/to/dir/file. We have many metrics implemented for it using the Metrics2 
interface. Tests are implemented mostly using a mock implementation for the 
Azure SDK functionality, with an option to test against a real blob storage if 
configured (instructions provided inside in README.txt).

h2. Credits and history
This has been ongoing work for a while, and the early version of this work can 
be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll 
post the patch here for Hadoop trunk first, then post a patch for branch-1 as 
well for backporting the functionality if accepted. Credit for this work goes 
to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as 
well as multiple people who have taken over this work since then (hope I don't 
forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, 
[~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu].

h2. Test
Besides unit tests, we have used WASB as the default file system in our service 
product. (HDFS is also used but not as default file system.) Various different 
customer and test workloads have been run against clusters with such 
configurations for quite some time. The current version reflects to the version 
of the WASB tested and used in our production environment.


 Support Windows Azure Storage - Blob as a file system in Hadoop
 ---

 Key: HADOOP-9629
 URL: https://issues.apache.org/jira/browse/HADOOP-9629
 Project: Hadoop Common
  

[jira] [Updated] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9629:
--

Summary: Support Windows Azure Storage - Blob as a file system in Hadoop  
(was: Support Azure Blob Storage as a file system in Hadoop)

 Support Windows Azure Storage - Blob as a file system in Hadoop
 ---

 Key: HADOOP-9629
 URL: https://issues.apache.org/jira/browse/HADOOP-9629
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
 Attachments: HADOOP-9629.2.patch, HADOOP-9629.patch


 h2. Description
 This JIRA incorporates adding a new file system implementation for accessing 
 Windows Azure Blob storage from within Hadoop, such as using blobs as input 
 to MR jobs or configuring MR jobs to put their output directly into blob 
 storage.
 h2. High level design
 At a high level, the code here extends the FileSystem class to provide an 
 implementation for accessing blob storage; the scheme asv is used for 
 accessing it over HTTP, and asvs for accessing over HTTPS. We use the URI 
 scheme: {code}asv[s]://container@account/path/to/file{code} to address 
 individual blobs. We use the standard Azure Java SDK 
 (com.microsoft.windowsazure) to do most of the work. In order to map a 
 hierarchical file system over the flat name-value pair nature of blob 
 storage, we create a specially tagged blob named path/to/dir whenever we 
 create a directory called path/to/dir, then files under that are stored as 
 normal blobs path/to/dir/file. We have many metrics implemented for it using 
 the Metrics2 interface. Tests are implemented mostly using a mock 
 implementation for the Azure SDK functionality, with an option to test 
 against a real blob storage if configured (instructions provided inside in 
 RunningLiveAsvTests.txt).
 h2. Credits and history
 This has been ongoing work for a while, and the early version of this work 
 can be seen in HADOOP-8079. This JIRA is a significant revision of that and 
 we'll post the patch here for Hadoop trunk first, then post a patch for 
 branch-1 as well for backporting the functionality if accepted. Credit for 
 this work goes to the early team: Min Wei, David Lao, Lengning Liu and 
 Alexander Stojanovic as well as multiple people who have taken over this work 
 since then (hope I don't forget anyone): Dexter Bradshaw, Johannes Klein, 
 Ivan Mitic, Michael Rys and Mostafa Elhemali.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9629:
--

Description: 
h2. Description
This JIRA incorporates adding a new file system implementation for accessing 
Windows Azure Storage - Blob from within Hadoop, such as using blobs as input 
to MR jobs or configuring MR jobs to put their output directly into blob 
storage.

h2. High level design
At a high level, the code here extends the FileSystem class to provide an 
implementation for accessing blob storage; the scheme wasb is used for 
accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI 
scheme: {code}wasb[s]://container@account/path/to/file{code} to address 
individual blobs. We use the standard Azure Java SDK 
(com.microsoft.windowsazure) to do most of the work. In order to map a 
hierarchical file system over the flat name-value pair nature of blob storage, 
we create a specially tagged blob named path/to/dir whenever we create a 
directory called path/to/dir, then files under that are stored as normal blobs 
path/to/dir/file. We have many metrics implemented for it using the Metrics2 
interface. Tests are implemented mostly using a mock implementation for the 
Azure SDK functionality, with an option to test against a real blob storage if 
configured (instructions provided inside in README.txt).

h2. Credits and history
This has been ongoing work for a while, and the early version of this work can 
be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll 
post the patch here for Hadoop trunk first, then post a patch for branch-1 as 
well for backporting the functionality if accepted. Credit for this work goes 
to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as 
well as multiple people who have taken over this work since then (hope I don't 
forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, 
[~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu].

h2. Test
Besides unit tests, we have used WASB as the default file system in our service 
product. (HDFS is also used but not as default file system.) Various different 
customer and test workloads have been run against clusters with such 
configurations for quite some time. The current version reflects to the version 
of the WASB tested and used in our production environment.

  was:
h2. Description
This JIRA incorporates adding a new file system implementation for accessing 
Windows Azure Blob storage from within Hadoop, such as using blobs as input to 
MR jobs or configuring MR jobs to put their output directly into blob storage.

h2. High level design
At a high level, the code here extends the FileSystem class to provide an 
implementation for accessing blob storage; the scheme asv is used for accessing 
it over HTTP, and asvs for accessing over HTTPS. We use the URI scheme: 
{code}asv[s]://container@account/path/to/file{code} to address individual 
blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do 
most of the work. In order to map a hierarchical file system over the flat 
name-value pair nature of blob storage, we create a specially tagged blob named 
path/to/dir whenever we create a directory called path/to/dir, then files under 
that are stored as normal blobs path/to/dir/file. We have many metrics 
implemented for it using the Metrics2 interface. Tests are implemented mostly 
using a mock implementation for the Azure SDK functionality, with an option to 
test against a real blob storage if configured (instructions provided inside in 
RunningLiveAsvTests.txt).

h2. Credits and history
This has been ongoing work for a while, and the early version of this work can 
be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll 
post the patch here for Hadoop trunk first, then post a patch for branch-1 as 
well for backporting the functionality if accepted. Credit for this work goes 
to the early team: Min Wei, David Lao, Lengning Liu and Alexander Stojanovic as 
well as multiple people who have taken over this work since then (hope I don't 
forget anyone): Dexter Bradshaw, Johannes Klein, Ivan Mitic, Michael Rys and 
Mostafa Elhemali.


 Support Windows Azure Storage - Blob as a file system in Hadoop
 ---

 Key: HADOOP-9629
 URL: https://issues.apache.org/jira/browse/HADOOP-9629
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
 Attachments: HADOOP-9629.2.patch, HADOOP-9629.patch


 h2. Description
 This JIRA incorporates adding a new file system implementation for accessing 
 Windows Azure Storage - Blob from within Hadoop, such as using blobs as input 
 to MR jobs or configuring MR jobs to put their output directly into blob 

[jira] [Updated] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9629:
--

Attachment: HADOOP-9629.3.patch

Attach a new patch. Slightly updated the description to address the schema 
change from {{asv(s)}} to {{wasb(s)}}. Adding to the creditors, [~mikelid], 
[~brian_swan], [~xifang], and myself.

 Support Windows Azure Storage - Blob as a file system in Hadoop
 ---

 Key: HADOOP-9629
 URL: https://issues.apache.org/jira/browse/HADOOP-9629
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
 Attachments: HADOOP-9629.2.patch, HADOOP-9629.3.patch, 
 HADOOP-9629.patch


 h2. Description
 This JIRA incorporates adding a new file system implementation for accessing 
 Windows Azure Storage - Blob from within Hadoop, such as using blobs as input 
 to MR jobs or configuring MR jobs to put their output directly into blob 
 storage.
 h2. High level design
 At a high level, the code here extends the FileSystem class to provide an 
 implementation for accessing blob storage; the scheme wasb is used for 
 accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI 
 scheme: {code}wasb[s]://container@account/path/to/file{code} to address 
 individual blobs. We use the standard Azure Java SDK 
 (com.microsoft.windowsazure) to do most of the work. In order to map a 
 hierarchical file system over the flat name-value pair nature of blob 
 storage, we create a specially tagged blob named path/to/dir whenever we 
 create a directory called path/to/dir, then files under that are stored as 
 normal blobs path/to/dir/file. We have many metrics implemented for it using 
 the Metrics2 interface. Tests are implemented mostly using a mock 
 implementation for the Azure SDK functionality, with an option to test 
 against a real blob storage if configured (instructions provided inside in 
 README.txt).
 h2. Credits and history
 This has been ongoing work for a while, and the early version of this work 
 can be seen in HADOOP-8079. This JIRA is a significant revision of that and 
 we'll post the patch here for Hadoop trunk first, then post a patch for 
 branch-1 as well for backporting the functionality if accepted. Credit for 
 this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and 
 [~stojanovic] as well as multiple people who have taken over this work since 
 then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], 
 Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and 
 [~chuanliu].
 h2. Test
 Besides unit tests, we have used WASB as the default file system in our 
 service product. (HDFS is also used but not as default file system.) Various 
 different customer and test workloads have been run against clusters with 
 such configurations for quite some time. The current version reflects to the 
 version of the WASB tested and used in our production environment.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HADOOP-10133) winutils detection on windows-cygwin fails

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13877037#comment-13877037
 ] 

Chuan Liu commented on HADOOP-10133:


Is HADOOP_HOME environment variable or Java property hadoop.home.dir been 
set? We are relying on these two methods to discover winutils.exe. Also 
winutils.exe have no dependency on cygwin, so there should be no need to 
start the Hadoop processes from cygwin.

 winutils detection on windows-cygwin fails
 --

 Key: HADOOP-10133
 URL: https://issues.apache.org/jira/browse/HADOOP-10133
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.2.0
 Environment: windows 7, cygwin
Reporter: Franjo Markovic
   Original Estimate: 1h
  Remaining Estimate: 1h

 java.io.IOException: Could not locate executable null\bin\winutils.exe in the 
 Hadoop binaries.
 at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
  at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
 at org.apache.hadoop.util.Shell.clinit(Shell.java:293)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HADOOP-10140) Specification of HADOOP_CONF_DIR via the environment in hadoop_config.cmd

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13877041#comment-13877041
 ] 

Chuan Liu commented on HADOOP-10140:


Hey [~ian_jack...@trilliumsoftware.com], I think HADOOP_CONF_DIR should be set 
as YARN_CONF_DIR from my understanding of the code. In most cmd files including 
yarn-conf.cmd, we will call hadoop-config.cmd. In hadoop-conf.cmd, we will set 
HADOOP_CONF_DIR to the command line argument of --config. Please let me know 
if you see it differently.

 Specification of HADOOP_CONF_DIR via the environment in hadoop_config.cmd
 -

 Key: HADOOP-10140
 URL: https://issues.apache.org/jira/browse/HADOOP-10140
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.2.0
 Environment: This Windows when using the Command processing as 
 opposed to using Bash from Cygwin.
Reporter: Ian Jackson
Priority: Minor

 It would be nice if HADOOP_CONF_DIR could be set in the environment like 
 YARN_CONF_DIR. This could be done in lib-exec/hadoop.cmd by setting 
 HADOOP_CONF_DIR conditionally.
 Using the Windows command shell, the modification would be as follows:
 if not defined HADOOP_CONF_DIR (
 set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop
 )
 This would allow the Hadoop configuration to be placed in ProgramData more 
 easily. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HADOOP-10217) Unable to run 'hadoop' commands, after installing on Cygwin

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13877050#comment-13877050
 ] 

Chuan Liu commented on HADOOP-10217:


I suggest just set HADOOP_HOME and JAVA_HOME using Windows path instead of 
cygwin path and run the commands from a Windows cmd shell instead of Windows 
shell. However, I have never tested the Hadoop on Windows XP; so I cannot 
guarantee it will work.

 Unable to run 'hadoop' commands, after installing on Cygwin
 ---

 Key: HADOOP-10217
 URL: https://issues.apache.org/jira/browse/HADOOP-10217
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Affects Versions: 2.2.0
 Environment: Installed on Cygwin (latest version) running Window XP. 
 Set Java 1.7.0_45 path (JDK) to /cygdrive/e/JDK. Installed ssh (on 
 /cygdrive/e/Openssh-6.4p1), created all keys and stored on 
 /home/admin.Installed hadoop-2.2.0 on /cygdrive/e/hadoop-2.2.0
Reporter: Anand Murali
  Labels: test

 Did following
 1. export JAVA_HOME=/cygdrive/e/JDK
 2. export HADOOP_INSTALL=/cygdrive/e/hadoop-2.2.0
 3. export 
 PATH=:$PATH:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin:$HADOOP_INSTALL/etc:$HADOOP_INSTALL/share:$HADOOP_INSTALL/lib:$HADOOP_INSTALL/libexec
 $hadoop version
 Error: Could not find or load main class org.apache.hadoop.util.VersionInfo.
 Cannot run anymore commands. I am unable to detect what path problems is 
 causing this error 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HADOOP-10178) Configuration deprecation always emit deprecated warnings when a new key is used

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863698#comment-13863698
 ] 

Chuan Liu commented on HADOOP-10178:


+1

The change looks good to me!

 Configuration deprecation always emit deprecated warnings when a new key is 
 used
 --

 Key: HADOOP-10178
 URL: https://issues.apache.org/jira/browse/HADOOP-10178
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Affects Versions: 2.2.0
Reporter: shanyu zhao
Assignee: shanyu zhao
 Attachments: HADOOP-10178-v2.patch, HADOOP-10178-v3.patch, 
 HADOOP-10178.patch


 Even if you use any new configuration properties, you still find deprecated 
 warnings in your logs. E.g.:
 13/12/14 01:00:51 INFO Configuration.deprecation: mapred.input.dir.recursive 
 is deprecated. Instead, use 
 mapreduce.input.fileinputformat.input.dir.recursive



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HADOOP-10175) Har files system authority should preserve userinfo

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10175:
---

Attachment: HADOOP-10175.2.patch

Thanks for reviewing, Chris! Attaching a new patch that corrects the typo.

 Har files system authority should preserve userinfo
 ---

 Key: HADOOP-10175
 URL: https://issues.apache.org/jira/browse/HADOOP-10175
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.2.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10175.2.patch, HADOOP-10175.patch


 When Har file system parse the URI to get the authority at initialization, 
 the userinfo is not preserved. This may lead to failures if the underlying 
 file system relies on the userinfo to work properly. E.g. 
 har://file-user:passwd@localhost:80/test.har will be parsed to 
 har://file-localhost:80/test.har, where user:passwd is lost in the processing.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10175) Har files system authority should preserve userinfo

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-10175:
--

 Summary: Har files system authority should preserve userinfo
 Key: HADOOP-10175
 URL: https://issues.apache.org/jira/browse/HADOOP-10175
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu


When Har file system parse the URI get the authority at initialization, the 
userinfo is not preserved. This may lead to failures if the underlying file 
system relies on the userinfo to work properly. E.g. 
har://file-user:passwd@localhost:80/test.har will be parsed to 
har://file-localhost:80/test.har, where user:passwd is lost in the processing.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HADOOP-10175) Har files system authority should preserve userinfo

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10175:
---

Status: Patch Available  (was: Open)

 Har files system authority should preserve userinfo
 ---

 Key: HADOOP-10175
 URL: https://issues.apache.org/jira/browse/HADOOP-10175
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10175.patch


 When Har file system parse the URI get the authority at initialization, the 
 userinfo is not preserved. This may lead to failures if the underlying file 
 system relies on the userinfo to work properly. E.g. 
 har://file-user:passwd@localhost:80/test.har will be parsed to 
 har://file-localhost:80/test.har, where user:passwd is lost in the processing.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HADOOP-10175) Har files system authority should preserve userinfo

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10175:
---

Attachment: HADOOP-10175.patch

Attaching a patch. A unit test is also added to cover the case.

 Har files system authority should preserve userinfo
 ---

 Key: HADOOP-10175
 URL: https://issues.apache.org/jira/browse/HADOOP-10175
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10175.patch


 When Har file system parse the URI get the authority at initialization, the 
 userinfo is not preserved. This may lead to failures if the underlying file 
 system relies on the userinfo to work properly. E.g. 
 har://file-user:passwd@localhost:80/test.har will be parsed to 
 har://file-localhost:80/test.har, where user:passwd is lost in the processing.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HADOOP-10175) Har files system authority should preserve userinfo

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10175:
---

Description: When Har file system parse the URI to get the authority at 
initialization, the userinfo is not preserved. This may lead to failures if the 
underlying file system relies on the userinfo to work properly. E.g. 
har://file-user:passwd@localhost:80/test.har will be parsed to 
har://file-localhost:80/test.har, where user:passwd is lost in the processing.  
(was: When Har file system parse the URI get the authority at initialization, 
the userinfo is not preserved. This may lead to failures if the underlying file 
system relies on the userinfo to work properly. E.g. 
har://file-user:passwd@localhost:80/test.har will be parsed to 
har://file-localhost:80/test.har, where user:passwd is lost in the processing.)

 Har files system authority should preserve userinfo
 ---

 Key: HADOOP-10175
 URL: https://issues.apache.org/jira/browse/HADOOP-10175
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10175.patch


 When Har file system parse the URI to get the authority at initialization, 
 the userinfo is not preserved. This may lead to failures if the underlying 
 file system relies on the userinfo to work properly. E.g. 
 har://file-user:passwd@localhost:80/test.har will be parsed to 
 har://file-localhost:80/test.har, where user:passwd is lost in the processing.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HADOOP-10110) hadoop-auth has a build break due to missing dependency

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-10110:
--

 Summary: hadoop-auth has a build break due to missing dependency
 Key: HADOOP-10110
 URL: https://issues.apache.org/jira/browse/HADOOP-10110
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Blocker


We have a build break in hadoop-auth if build with maven cache cleaned. The 
error looks like the follows. The problem exists on both Windows and Linux. If 
you have old jetty jars in your maven cache, you won't see the error.

{noformat}
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:29.469s
[INFO] Finished at: Mon Nov 18 12:30:36 PST 2013
[INFO] Final Memory: 37M/120M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
(default-testCompile) on project hadoop-auth: Compilation failure: Compilation 
failure:
[ERROR] 
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
 cannot access org.mortbay.component.AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
[ERROR] server = new Server(0);
[ERROR] 
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[94,29]
 cannot access org.mortbay.component.LifeCycle
[ERROR] class file for org.mortbay.component.LifeCycle not found
[ERROR] server.getConnectors()[0].setHost(host);
[ERROR] 
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,10]
 cannot find symbol
[ERROR] symbol  : method start()
[ERROR] location: class org.mortbay.jetty.Server
[ERROR] 
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[102,12]
 cannot find symbol
[ERROR] symbol  : method stop()
[ERROR] location: class org.mortbay.jetty.Server
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn goals -rf :hadoop-auth

{noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10110) hadoop-auth has a build break due to missing dependency

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10110:
---

Status: Patch Available  (was: Open)

 hadoop-auth has a build break due to missing dependency
 ---

 Key: HADOOP-10110
 URL: https://issues.apache.org/jira/browse/HADOOP-10110
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Blocker
 Attachments: HADOOP-10110.patch


 We have a build break in hadoop-auth if build with maven cache cleaned. The 
 error looks like the follows. The problem exists on both Windows and Linux. 
 If you have old jetty jars in your maven cache, you won't see the error.
 {noformat}
 [INFO] 
 
 [INFO] BUILD FAILURE
 [INFO] 
 
 [INFO] Total time: 1:29.469s
 [INFO] Finished at: Mon Nov 18 12:30:36 PST 2013
 [INFO] Final Memory: 37M/120M
 [INFO] 
 
 [ERROR] Failed to execute goal 
 org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
 (default-testCompile) on project hadoop-auth: Compilation failure: 
 Compilation failure:
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
  cannot access org.mortbay.component.AbstractLifeCycle
 [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
 [ERROR] server = new Server(0);
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[94,29]
  cannot access org.mortbay.component.LifeCycle
 [ERROR] class file for org.mortbay.component.LifeCycle not found
 [ERROR] server.getConnectors()[0].setHost(host);
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,10]
  cannot find symbol
 [ERROR] symbol  : method start()
 [ERROR] location: class org.mortbay.jetty.Server
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[102,12]
  cannot find symbol
 [ERROR] symbol  : method stop()
 [ERROR] location: class org.mortbay.jetty.Server
 [ERROR] - [Help 1]
 [ERROR]
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
 switch.
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR]
 [ERROR] For more information about the errors and possible solutions, please 
 read the following articles:
 [ERROR] [Help 1] 
 http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
 [ERROR]
 [ERROR] After correcting the problems, you can resume the build with the 
 command
 [ERROR]   mvn goals -rf :hadoop-auth
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10110) hadoop-auth has a build break due to missing dependency

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10110:
---

Affects Version/s: 3.0.0

 hadoop-auth has a build break due to missing dependency
 ---

 Key: HADOOP-10110
 URL: https://issues.apache.org/jira/browse/HADOOP-10110
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Blocker
 Attachments: HADOOP-10110.patch


 We have a build break in hadoop-auth if build with maven cache cleaned. The 
 error looks like the follows. The problem exists on both Windows and Linux. 
 If you have old jetty jars in your maven cache, you won't see the error.
 {noformat}
 [INFO] 
 
 [INFO] BUILD FAILURE
 [INFO] 
 
 [INFO] Total time: 1:29.469s
 [INFO] Finished at: Mon Nov 18 12:30:36 PST 2013
 [INFO] Final Memory: 37M/120M
 [INFO] 
 
 [ERROR] Failed to execute goal 
 org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
 (default-testCompile) on project hadoop-auth: Compilation failure: 
 Compilation failure:
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
  cannot access org.mortbay.component.AbstractLifeCycle
 [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
 [ERROR] server = new Server(0);
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[94,29]
  cannot access org.mortbay.component.LifeCycle
 [ERROR] class file for org.mortbay.component.LifeCycle not found
 [ERROR] server.getConnectors()[0].setHost(host);
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,10]
  cannot find symbol
 [ERROR] symbol  : method start()
 [ERROR] location: class org.mortbay.jetty.Server
 [ERROR] 
 /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[102,12]
  cannot find symbol
 [ERROR] symbol  : method stop()
 [ERROR] location: class org.mortbay.jetty.Server
 [ERROR] - [Help 1]
 [ERROR]
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
 switch.
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR]
 [ERROR] For more information about the errors and possible solutions, please 
 read the following articles:
 [ERROR] [Help 1] 
 http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
 [ERROR]
 [ERROR] After correcting the problems, you can resume the build with the 
 command
 [ERROR]   mvn goals -rf :hadoop-auth
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HADOOP-10082) TestHDFSCLI fails on Windows due to difference in line endings

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-10082:
--

 Summary: TestHDFSCLI fails on Windows due to difference in line 
endings
 Key: HADOOP-10082
 URL: https://issues.apache.org/jira/browse/HADOOP-10082
 Project: Hadoop Common
  Issue Type: Test
Affects Versions: 3.0.0
Reporter: Chuan Liu
Priority: Minor


This is similar issue with HADOOP-8657. git appended /r by default on Windows 
at checkout. When TestHDFSCLI runs, the verification will fail on file size for 
files like data15bytes, data30bytes, etc.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10031) FsShell -get/copyFromLocal/moveFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10031:
---

Summary: FsShell -get/copyFromLocal/moveFromLocal should support Windows 
local path  (was: Investigate PathData.expandAsGlob() support for Windows path)

 FsShell -get/copyFromLocal/moveFromLocal should support Windows local path
 --

 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu

 The current PathData.expandAsGlob() does not support Windows path of the 
 pattern C:\some\dir. This method is used by many fs shell command to expand 
 the paths from command line arguments. As a result, running some fs shell 
 command with Windows local path as arguments will fail. This is a regression 
 with 1.0.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10031:
---

Description: In current trunk code, running 'hadoop fs -get /hdfs/path 
c:\windows\path' will lead to an error put: unexpected URISyntaxException. This 
is a regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and 
moveFromLocal should support Windows local path formats as localSrc argument to 
the two commands.  (was: In current trunk code, running 'hadoop fs -get 
c:\windows\path /hdfs/path' will lead to an error put: unexpected 
URISyntaxException. This is a regression to the 1.0 fs shell commands. FsShell 
put and copyFromLocal should support Windows local path formats as localSrc 
argument to the two commands. This is a regression with 1.0.)

 FsShell -get/copyToLocal/moveFromLocal should support Windows local path
 

 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu

 In current trunk code, running 'hadoop fs -get /hdfs/path c:\windows\path' 
 will lead to an error put: unexpected URISyntaxException. This is a 
 regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and 
 moveFromLocal should support Windows local path formats as localSrc argument 
 to the two commands.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10031:
---

Summary: FsShell -get/copyToLocal/moveFromLocal should support Windows 
local path  (was: FsShell -get/copyFromLocal/moveFromLocal should support 
Windows local path)

 FsShell -get/copyToLocal/moveFromLocal should support Windows local path
 

 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu

 In current trunk code, running 'hadoop fs -get c:\windows\path /hdfs/path' 
 will lead to an error put: unexpected URISyntaxException. This is a 
 regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should 
 support Windows local path formats as localSrc argument to the two commands. 
 This is a regression with 1.0.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10031) FsShell -get/copyFromLocal/moveFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10031:
---

Description: In current trunk code, running 'hadoop fs -get c:\windows\path 
/hdfs/path' will lead to an error put: unexpected URISyntaxException. This is a 
regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should 
support Windows local path formats as localSrc argument to the two commands. 
This is a regression with 1.0.  (was: The current PathData.expandAsGlob() does 
not support Windows path of the pattern C:\some\dir. This method is used by 
many fs shell command to expand the paths from command line arguments. As a 
result, running some fs shell command with Windows local path as arguments will 
fail. This is a regression with 1.0.)

 FsShell -get/copyFromLocal/moveFromLocal should support Windows local path
 --

 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu

 In current trunk code, running 'hadoop fs -get c:\windows\path /hdfs/path' 
 will lead to an error put: unexpected URISyntaxException. This is a 
 regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should 
 support Windows local path formats as localSrc argument to the two commands. 
 This is a regression with 1.0.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10031:
---

Attachment: HADOOP-10031.patch

Attach a patch. I added two unit tests. The -moveFromLocal was actually fixed 
by HADOOP-10030. Adding the unit test to ensure no regression in the future.

 FsShell -get/copyToLocal/moveFromLocal should support Windows local path
 

 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10031.patch


 In current trunk code, running 'hadoop fs -get /hdfs/path c:\windows\path' 
 will lead to an error put: unexpected URISyntaxException. This is a 
 regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and 
 moveFromLocal should support Windows local path formats as localSrc argument 
 to the two commands.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10031:
---

Status: Patch Available  (was: Open)

 FsShell -get/copyToLocal/moveFromLocal should support Windows local path
 

 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10031.patch


 In current trunk code, running 'hadoop fs -get /hdfs/path c:\windows\path' 
 will lead to an error put: unexpected URISyntaxException. This is a 
 regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and 
 moveFromLocal should support Windows local path formats as localSrc argument 
 to the two commands.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10031:
---

Attachment: HADOOP-10031.1.patch

Attach a new patch. Fix the unit test case name.

 FsShell -get/copyToLocal/moveFromLocal should support Windows local path
 

 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10031.1.patch, HADOOP-10031.patch


 In current trunk code, running 'hadoop fs -get /hdfs/path c:\windows\path' 
 will lead to an error put: unexpected URISyntaxException. This is a 
 regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and 
 moveFromLocal should support Windows local path formats as localSrc argument 
 to the two commands.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HADOOP-10030) FsShell -put/copyFromLocal should support Windows local path

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-10030:
--

 Summary: FsShell -put/copyFromLocal should support Windows local 
path
 Key: HADOOP-10030
 URL: https://issues.apache.org/jira/browse/HADOOP-10030
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu


In current trunk code, running 'hadoop fs -put c:\windows\path /hdfs/path' will 
lead to an error {{put: unexpected URISyntaxException}}. This is a regression 
to the 1.0 fs shell commands. FsShell put and copyFromLocal should support 
Windows local path formats as localSrc argument to the two commands.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10030) FsShell -put/copyFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10030:
---

Attachment: HADOOP-10030.patch

Attaching a patch. I also added two unit tests for the scenario by reusing some 
existing test methods.

 FsShell -put/copyFromLocal should support Windows local path
 

 Key: HADOOP-10030
 URL: https://issues.apache.org/jira/browse/HADOOP-10030
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10030.patch


 In current trunk code, running 'hadoop fs -put c:\windows\path /hdfs/path' 
 will lead to an error {{put: unexpected URISyntaxException}}. This is a 
 regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should 
 support Windows local path formats as localSrc argument to the two commands.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-10030) FsShell -put/copyFromLocal should support Windows local path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-10030:
---

Status: Patch Available  (was: Open)

 FsShell -put/copyFromLocal should support Windows local path
 

 Key: HADOOP-10030
 URL: https://issues.apache.org/jira/browse/HADOOP-10030
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-10030.patch


 In current trunk code, running 'hadoop fs -put c:\windows\path /hdfs/path' 
 will lead to an error {{put: unexpected URISyntaxException}}. This is a 
 regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should 
 support Windows local path formats as localSrc argument to the two commands.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HADOOP-10031) Investigate PathData.expandAsGlob() support for Windows path

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-10031:
--

 Summary: Investigate PathData.expandAsGlob() support for Windows 
path
 Key: HADOOP-10031
 URL: https://issues.apache.org/jira/browse/HADOOP-10031
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu


The current PathData.expandAsGlob() does not support Windows path of the 
pattern C:\some\dir. This method is used by many fs shell command to expand 
the paths from command line arguments. As a result, running some fs shell 
command with Windows local path as arguments will fail. This is a regression 
with 1.0.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HADOOP-9948) Add a config value to CLITestHelper to skip tests on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9948:
--

Issue Type: Test  (was: Bug)

 Add a config value to CLITestHelper to skip tests on Windows
 

 Key: HADOOP-9948
 URL: https://issues.apache.org/jira/browse/HADOOP-9948
 Project: Hadoop Common
  Issue Type: Test
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9948.patch


 We want to add a configuration to CLITestHelper so we can skip some tests on 
 Windows. We want this feature because some test cases in TestHDFSCLI should 
 not be tested on Windows. Or more specifically, the globbing test cases. The 
 differences are explained in HDFS-4632. The proposed syntax looks like 
 follows.
 {noformat}
 test !-- TESTED --
   descriptionls: Negative test for quoted /*/* globbing /description
   windowsfalse/windows
 {noformat}
 When Windows set to false, we will skip running the test on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9948) Add a config value to CLITestHelper to skip tests on Windows

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-9948:
-

 Summary: Add a config value to CLITestHelper to skip tests on 
Windows
 Key: HADOOP-9948
 URL: https://issues.apache.org/jira/browse/HADOOP-9948
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor


We want to add a configuration to CLITestHelper so we can skip some tests on 
Windows. We want this feature because some test cases in TestHDFSCLI should not 
be tested on Windows. Or more specifically, the globbing test cases. The 
differences are explained in HDFS-4632. The proposed syntax looks like follows.

{noformat}
test !-- TESTED --
  descriptionls: Negative test for quoted /*/* globbing /description
  windowsfalse/windows
{noformat}

When Windows set to false, we will skip running the test on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9948) Add a config value to CLITestHelper to skip tests on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9948:
--

Attachment: HADOOP-9948.patch

Attach a patch. Tested on Linux and Windows. The three test cases skipped on 
Windows still run on Linux.

 Add a config value to CLITestHelper to skip tests on Windows
 

 Key: HADOOP-9948
 URL: https://issues.apache.org/jira/browse/HADOOP-9948
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9948.patch


 We want to add a configuration to CLITestHelper so we can skip some tests on 
 Windows. We want this feature because some test cases in TestHDFSCLI should 
 not be tested on Windows. Or more specifically, the globbing test cases. The 
 differences are explained in HDFS-4632. The proposed syntax looks like 
 follows.
 {noformat}
 test !-- TESTED --
   descriptionls: Negative test for quoted /*/* globbing /description
   windowsfalse/windows
 {noformat}
 When Windows set to false, we will skip running the test on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9948) Add a config value to CLITestHelper to skip tests on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9948:
--

Status: Patch Available  (was: Open)

 Add a config value to CLITestHelper to skip tests on Windows
 

 Key: HADOOP-9948
 URL: https://issues.apache.org/jira/browse/HADOOP-9948
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9948.patch


 We want to add a configuration to CLITestHelper so we can skip some tests on 
 Windows. We want this feature because some test cases in TestHDFSCLI should 
 not be tested on Windows. Or more specifically, the globbing test cases. The 
 differences are explained in HDFS-4632. The proposed syntax looks like 
 follows.
 {noformat}
 test !-- TESTED --
   descriptionls: Negative test for quoted /*/* globbing /description
   windowsfalse/windows
 {noformat}
 When Windows set to false, we will skip running the test on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9896) TestIPC fail on trunk with error VM crash or System.exit

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9896:
--

  Resolution: Fixed
Release Note: Resolving this as HADOOP-9916 should fix the root cause in 
IPC Client implementation.
  Status: Resolved  (was: Patch Available)

 TestIPC fail on trunk with error VM crash or System.exit
 

 Key: HADOOP-9896
 URL: https://issues.apache.org/jira/browse/HADOOP-9896
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 3.0.0, 2.3.0
Reporter: shanyu zhao
Assignee: Chuan Liu
 Attachments: HADOOP-9896.2.patch, HADOOP-9896.patch, 
 org.apache.hadoop.ipc.TestIPC-output.txt


 I'm running hadoop unit tests on a Ubuntu 12.04 64 bit virtual machine, every 
 time I try to run all unit tests with command mvn test, the TestIPC unit 
 test will fail, the console will show The forked VM terminated without 
 saying properly goodbye. VM crash or System.exit called?
 To reproduce:
 $cd hadoop-common-project/hadoop-common
 $mvn clean install -Pdist -DskipTests
 $mvn test -Pdist -Dtest=TestIPC

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9924) FileUtil.createJarWithClassPath() does not generate relative classpath correctly

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757036#comment-13757036
 ] 

Chuan Liu commented on HADOOP-9924:
---

+1 from me as well.

bq. -1 javac. The patch appears to cause the build to fail.

I think you may need to remove Windows new line characters for the patch to 
apply. Easiest way may be to run 'dos2unix.exe' on the patch.

 FileUtil.createJarWithClassPath() does not generate relative classpath 
 correctly
 

 Key: HADOOP-9924
 URL: https://issues.apache.org/jira/browse/HADOOP-9924
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.1.0-beta, 0.23.9
Reporter: shanyu zhao
Assignee: shanyu zhao
 Attachments: HADOOP-9924-2.patch, HADOOP-9924-3.patch, 
 HADOOP-9924.patch


 On Windows, FileUtil.createJarWithClassPath() is called to generate a 
 manifest jar file to pack classpath - to avoid the problem of classpath being 
 too long.
 However, the relative classpath is not handled correctly. It relies on Java's 
 File(relativePath) to resolve the relative path. But it really should be 
 using the given pwd parameter to resolve the relative path.
 To reproduce this bug, you can try some pig job on Windows, it will fail and 
 the pig log on the application master will look like this:
 2013-08-29 23:25:55,498 INFO [main] 
 org.apache.hadoop.service.AbstractService: Service 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: 
 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
 java.lang.RuntimeException: java.lang.ClassNotFoundException: Class 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat 
 not found
 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
 java.lang.RuntimeException: java.lang.ClassNotFoundException: Class 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat 
 not found
 This is because the PigOutputFormat class is in the job.jar file but the 
 classpath manifest has:
 file:/c:/apps/dist/hadoop-2.1.0-beta/bin/job.jar/job.jar
 When it really should be:
 file:/job container folder/job.jar/job.jar

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9916) Race condition in ipc.Client causes TestIPC timeout

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13756263#comment-13756263
 ] 

Chuan Liu commented on HADOOP-9916:
---

This is a very good analysis of the race condition.
I can verify that the patch also fixes the problem in HADOOP-9896.

 Race condition in ipc.Client causes TestIPC timeout
 ---

 Key: HADOOP-9916
 URL: https://issues.apache.org/jira/browse/HADOOP-9916
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Minor
 Attachments: HADOOP-9916.v1.patch


 TestIPC timeouts occasionally, for example: 
 [https://issues.apache.org/jira/browse/HDFS-5130?focusedCommentId=13749870page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13749870]
 [https://issues.apache.org/jira/browse/HADOOP-9915?focusedCommentId=13753302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13753302]
 Look into the code, there is race condition in oah.ipc.Client, the race 
 condition happen between RPC call thread and connection read response thread:
 {code}
 if (status == RpcStatusProto.SUCCESS) {
   Writable value = ReflectionUtils.newInstance(valueClass, conf);
   value.readFields(in); // read value
   call.setRpcResponse(value);
   calls.remove(callId);
 {code}
 Read Thread: 
 Connection.receiveRpcResponse- call.setRpcResponse(value) - notify Call 
 Thread
 Call Thread:
 Client.call - Connection.addCall(retry with the same callId) - notify read 
 thread
 Read Thread:
 calls.remove(callId) # intend to remove old call, but removes newly added 
 call...
 Connection.waitForWork end up wait maxIdleTime and close the connection. The 
 call never get response and dead.
 The problem doesn't show because previously callId is unique, we never 
 accidentally remove newly added calls, but when retry added this race 
 condition became possible.
 To solve this, we can simply change order, remove the call first, then notify 
 call thread.
 Note there are many places need this order change(normal case, error case, 
 cleanup case)
 And there are some minor issues in TestIPC:
 1. there are two method with same name:
void testSerial()
void testSerial(int handlerCount, boolean handlerSleep, ...)
the second is not a test case(so should not use testXXX prefix), but 
 somehow it causes testSerial(first one) run two times, see test report:
 {code}
   testcase time=26.896 classname=org.apache.hadoop.ipc.TestIPC 
 name=testSerial/
   testcase time=25.426 classname=org.apache.hadoop.ipc.TestIPC 
 name=testSerial/
 {code}
 2. timeout annotation should be added, so next time related log is available.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9896) TestIPC fail on trunk with error VM crash or System.exit

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13755630#comment-13755630
 ] 

Chuan Liu commented on HADOOP-9896:
---

[~macrokigol], I think the timeout is a separate issue. In our case, if we run 
testRetryProxy alone, it never fails. However when running all the test cases 
in the test class together, we will always get the JVM crash error due to 
testRetryProxy hangs.

It could be a race condition. What is your OS and configuration? [~shanyu] 
suspects your guys never run into our problem because you have faster machines. 
So it could also be that our Linux VM is slower and we never run into this 
timeout issue.

 TestIPC fail on trunk with error VM crash or System.exit
 

 Key: HADOOP-9896
 URL: https://issues.apache.org/jira/browse/HADOOP-9896
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 3.0.0, 2.3.0
Reporter: shanyu zhao
Assignee: Chuan Liu
 Attachments: HADOOP-9896.2.patch, HADOOP-9896.patch, 
 org.apache.hadoop.ipc.TestIPC-output.txt


 I'm running hadoop unit tests on a Ubuntu 12.04 64 bit virtual machine, every 
 time I try to run all unit tests with command mvn test, the TestIPC unit 
 test will fail, the console will show The forked VM terminated without 
 saying properly goodbye. VM crash or System.exit called?
 To reproduce:
 $cd hadoop-common-project/hadoop-common
 $mvn clean install -Pdist -DskipTests
 $mvn test -Pdist -Dtest=TestIPC

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9896) TestIPC fail on trunk with error VM crash or System.exit

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9896:
--

Assignee: Chuan Liu

 TestIPC fail on trunk with error VM crash or System.exit
 

 Key: HADOOP-9896
 URL: https://issues.apache.org/jira/browse/HADOOP-9896
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 2.0.5-alpha
Reporter: shanyu zhao
Assignee: Chuan Liu
 Attachments: org.apache.hadoop.ipc.TestIPC-output.txt


 I'm running hadoop unit tests on a Ubuntu 12.04 64 bit virtual machine, every 
 time I try to run all unit tests with command mvn test, the TestIPC unit 
 test will fail, the console will show The forked VM terminated without 
 saying properly goodbye. VM crash or System.exit called?
 To reproduce:
 $cd hadoop-common-project/hadoop-common
 $mvn clean install -Pdist -DskipTests
 $mvn test -Pdist -Dtest=TestIPC

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9896) TestIPC fail on trunk with error VM crash or System.exit

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9896:
--

Affects Version/s: (was: 2.0.5-alpha)
   2.3.0
   3.0.0
   Status: Patch Available  (was: Open)

 TestIPC fail on trunk with error VM crash or System.exit
 

 Key: HADOOP-9896
 URL: https://issues.apache.org/jira/browse/HADOOP-9896
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 3.0.0, 2.3.0
Reporter: shanyu zhao
Assignee: Chuan Liu
 Attachments: HADOOP-9896.patch, 
 org.apache.hadoop.ipc.TestIPC-output.txt


 I'm running hadoop unit tests on a Ubuntu 12.04 64 bit virtual machine, every 
 time I try to run all unit tests with command mvn test, the TestIPC unit 
 test will fail, the console will show The forked VM terminated without 
 saying properly goodbye. VM crash or System.exit called?
 To reproduce:
 $cd hadoop-common-project/hadoop-common
 $mvn clean install -Pdist -DskipTests
 $mvn test -Pdist -Dtest=TestIPC

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9896) TestIPC fail on trunk with error VM crash or System.exit

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9896:
--

Attachment: HADOOP-9896.patch

I investigated this problem a little bit. I think there is some socket leak in 
the test code. Attach a patch that fix the leaking issue in the test by 
explicitly closing client and server at the end of each test case. 

 TestIPC fail on trunk with error VM crash or System.exit
 

 Key: HADOOP-9896
 URL: https://issues.apache.org/jira/browse/HADOOP-9896
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 2.0.5-alpha
Reporter: shanyu zhao
Assignee: Chuan Liu
 Attachments: HADOOP-9896.patch, 
 org.apache.hadoop.ipc.TestIPC-output.txt


 I'm running hadoop unit tests on a Ubuntu 12.04 64 bit virtual machine, every 
 time I try to run all unit tests with command mvn test, the TestIPC unit 
 test will fail, the console will show The forked VM terminated without 
 saying properly goodbye. VM crash or System.exit called?
 To reproduce:
 $cd hadoop-common-project/hadoop-common
 $mvn clean install -Pdist -DskipTests
 $mvn test -Pdist -Dtest=TestIPC

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9896) TestIPC fail on trunk with error VM crash or System.exit

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13755196#comment-13755196
 ] 

Chuan Liu commented on HADOOP-9896:
---

bq.-1 core tests. The patch failed these unit tests in 
hadoop-common-project/hadoop-common:
bq.org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl

This failure does not seem related to the fix at all.

[~jingzhao], I will add timeout as suggested.

 TestIPC fail on trunk with error VM crash or System.exit
 

 Key: HADOOP-9896
 URL: https://issues.apache.org/jira/browse/HADOOP-9896
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 3.0.0, 2.3.0
Reporter: shanyu zhao
Assignee: Chuan Liu
 Attachments: HADOOP-9896.patch, 
 org.apache.hadoop.ipc.TestIPC-output.txt


 I'm running hadoop unit tests on a Ubuntu 12.04 64 bit virtual machine, every 
 time I try to run all unit tests with command mvn test, the TestIPC unit 
 test will fail, the console will show The forked VM terminated without 
 saying properly goodbye. VM crash or System.exit called?
 To reproduce:
 $cd hadoop-common-project/hadoop-common
 $mvn clean install -Pdist -DskipTests
 $mvn test -Pdist -Dtest=TestIPC

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9896) TestIPC fail on trunk with error VM crash or System.exit

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9896:
--

Attachment: HADOOP-9896.2.patch

Attach a new patch. A 30 sec timeout is added to each test case. The unit test 
can pass consistently on my single core Ubuntu box. So I assume the timeout is 
enough for the test cases.

 TestIPC fail on trunk with error VM crash or System.exit
 

 Key: HADOOP-9896
 URL: https://issues.apache.org/jira/browse/HADOOP-9896
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 3.0.0, 2.3.0
Reporter: shanyu zhao
Assignee: Chuan Liu
 Attachments: HADOOP-9896.2.patch, HADOOP-9896.patch, 
 org.apache.hadoop.ipc.TestIPC-output.txt


 I'm running hadoop unit tests on a Ubuntu 12.04 64 bit virtual machine, every 
 time I try to run all unit tests with command mvn test, the TestIPC unit 
 test will fail, the console will show The forked VM terminated without 
 saying properly goodbye. VM crash or System.exit called?
 To reproduce:
 $cd hadoop-common-project/hadoop-common
 $mvn clean install -Pdist -DskipTests
 $mvn test -Pdist -Dtest=TestIPC

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9887) globStatus does not correctly handle paths starting with a drive spec on Windows

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746968#comment-13746968
 ] 

Chuan Liu commented on HADOOP-9887:
---

bq. do you think it would make more sense to fix this in 
Globber#getPathComponents, conditional on whether or not the pattern was 
absolute like in the old FileSystem#globStatusInternal?

One problem is that we will lost the drive letter from the components; and we 
will list directories from / instead of /C:/. This has two effects: first, 
we could list from a wrong drive if the path passed in is not on the current 
drive; second, the result will be paths of the pattern without the drive 
letter. So I would still prefer the current approach. I can modify the comments 
to make it better fit the {{Globber#getPathComponents}} method. What do you 
think?

 globStatus does not correctly handle paths starting with a drive spec on 
 Windows
 

 Key: HADOOP-9887
 URL: https://issues.apache.org/jira/browse/HADOOP-9887
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.3.0
Reporter: Chris Nauroth
Assignee: Chuan Liu
 Attachments: HADOOP-9887-trunk.patch


 Recent file system changes have caused globStatus to stop working for paths 
 starting with a drive spec on Windows.  The problem is most easily visible by 
 running {{TestFileUtil#createJarWithClassPath}} on Windows.  This method 
 attempts a globStatus with pattern {{*\{.jar,.JAR\}}}, and it no longer 
 correctly identifies files at the path ending in .jar or .JAR.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9887) globStatus does not correctly handle paths starting with a drive spec on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9887:
--

Attachment: HADOOP-9887-trunk.2.patch

Thanks for the suggestion, Chris! Attach a new patch.

 globStatus does not correctly handle paths starting with a drive spec on 
 Windows
 

 Key: HADOOP-9887
 URL: https://issues.apache.org/jira/browse/HADOOP-9887
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.3.0
Reporter: Chris Nauroth
Assignee: Chuan Liu
 Attachments: HADOOP-9887-trunk.2.patch, HADOOP-9887-trunk.patch


 Recent file system changes have caused globStatus to stop working for paths 
 starting with a drive spec on Windows.  The problem is most easily visible by 
 running {{TestFileUtil#createJarWithClassPath}} on Windows.  This method 
 attempts a globStatus with pattern {{*\{.jar,.JAR\}}}, and it no longer 
 correctly identifies files at the path ending in .jar or .JAR.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9887) globStatus does not correctly handle paths starting with a drive spec on Windows

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13745158#comment-13745158
 ] 

Chuan Liu commented on HADOOP-9887:
---

I did some initial investigation on the issue.

I think the patch in HADOOP-9817 did not have the following check in the 
original {{FileSystem#globStatusInternal()}} method. The new relevant code is 
in {{Globber#glob()}}. As a result, Windows paths beginning with a drive letter 
no longer work with globStatus() method.

{code:java}
// determine starting point
int level = 0;
String baseDir = Path.CUR_DIR;
if (pathPattern.isAbsolute()) {
  level = 1; // need to skip empty item at beginning of split list
  baseDir = Path.SEPARATOR;
}
{code}

I am not sure why we need the above check for Unix in original 
{{FileSystem#globStatusInternal()}} method. However, it does help to skip the 
drive letter that may appear in a Windows path. Without the equivalent logic, 
the code follows will fail for a path that begins with drive letter. 1) We 
divide the path into components separated by path separator; 2) we begin list 
from root, and add file/directory that matches the component for each 
component in the components list to the candidate list. For path begins 
with drive letter, the code will think the drive letter as some path on root 
directory; thus leads to the failure.

 globStatus does not correctly handle paths starting with a drive spec on 
 Windows
 

 Key: HADOOP-9887
 URL: https://issues.apache.org/jira/browse/HADOOP-9887
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.3.0
Reporter: Chris Nauroth

 Recent file system changes have caused globStatus to stop working for paths 
 starting with a drive spec on Windows.  The problem is most easily visible by 
 running {{TestFileUtil#createJarWithClassPath}} on Windows.  This method 
 attempts a globStatus with pattern {{*\{.jar,.JAR\}}}, and it no longer 
 correctly identifies files at the path ending in .jar or .JAR.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9887) globStatus does not correctly handle paths starting with a drive spec on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9887:
--

Attachment: HADOOP-9887-trunk.patch

Attaching a patch. I will run all the unit tests on Windows to verify the patch.

 globStatus does not correctly handle paths starting with a drive spec on 
 Windows
 

 Key: HADOOP-9887
 URL: https://issues.apache.org/jira/browse/HADOOP-9887
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.3.0
Reporter: Chris Nauroth
 Attachments: HADOOP-9887-trunk.patch


 Recent file system changes have caused globStatus to stop working for paths 
 starting with a drive spec on Windows.  The problem is most easily visible by 
 running {{TestFileUtil#createJarWithClassPath}} on Windows.  This method 
 attempts a globStatus with pattern {{*\{.jar,.JAR\}}}, and it no longer 
 correctly identifies files at the path ending in .jar or .JAR.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9865:
--

Attachment: HADOOP-9865-trunk.patch

Attaching a patch. The fix looks exactly as Colin shows. So the credit also 
goes to him. I added a new test case in TestGlobPaths. Without the fix, the 
test fails with {{FileContext.globStatus()}} (this bug), and passes with 
{{FileSystem.globStatus()}}. I removed a previous pTestRelativePath test case 
because it is duplicated with the new test case I added. I tested this on 
Linux. On Windows, I need to incorporate the change in HDFS-5093 to get the 
test case pass. 

 FileContext.globStatus() has a regression with respect to relative path
 ---

 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.patch


 I discovered the problem when running unit test TestMRJobClient on Windows. 
 The cause is indirect in this case. In the unit test, we try to launch a job 
 and list its status. The job failed, and caused the list command get a result 
 of 0, which triggered the unit test assert. From the log and debug, the job 
 failed because we failed to create the Jar with classpath (see code around 
 {{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a 
 Windows specific step right now; so the test still passes on Linux. This step 
 failed because we passed in a relative path to {{FileContext.globStatus()}} 
 in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the 
 following.
 {noformat}
 2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] 
 launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch 
 container.
 org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
   at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
   at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
   at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
   at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
   at 
 org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
   at 
 org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {noformat}
 I think this is a regression from HADOOP-9817. I modified some code and the 
 unit test passed. (See the attached patch.) However, I think the impact is 
 larger. I will add some unit tests to verify the behavior, and work on a more 
 complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9865:
--

Attachment: HADOOP-9865-trunk.2.patch

Thanks for the suggestion! Attaching a new patch.

 FileContext.globStatus() has a regression with respect to relative path
 ---

 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch, 
 HADOOP-9865-trunk.patch


 I discovered the problem when running unit test TestMRJobClient on Windows. 
 The cause is indirect in this case. In the unit test, we try to launch a job 
 and list its status. The job failed, and caused the list command get a result 
 of 0, which triggered the unit test assert. From the log and debug, the job 
 failed because we failed to create the Jar with classpath (see code around 
 {{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a 
 Windows specific step right now; so the test still passes on Linux. This step 
 failed because we passed in a relative path to {{FileContext.globStatus()}} 
 in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the 
 following.
 {noformat}
 2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] 
 launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch 
 container.
 org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
   at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
   at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
   at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
   at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
   at 
 org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
   at 
 org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {noformat}
 I think this is a regression from HADOOP-9817. I modified some code and the 
 unit test passed. (See the attached patch.) However, I think the impact is 
 larger. I will add some unit tests to verify the behavior, and work on a more 
 complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9865:
--

Attachment: HADOOP-9865-trunk.3.patch

Attach a new version with some slight modification.

 FileContext.globStatus() has a regression with respect to relative path
 ---

 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch, 
 HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch


 I discovered the problem when running unit test TestMRJobClient on Windows. 
 The cause is indirect in this case. In the unit test, we try to launch a job 
 and list its status. The job failed, and caused the list command get a result 
 of 0, which triggered the unit test assert. From the log and debug, the job 
 failed because we failed to create the Jar with classpath (see code around 
 {{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a 
 Windows specific step right now; so the test still passes on Linux. This step 
 failed because we passed in a relative path to {{FileContext.globStatus()}} 
 in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the 
 following.
 {noformat}
 2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] 
 launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch 
 container.
 org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
   at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
   at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
   at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
   at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
   at 
 org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
   at 
 org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {noformat}
 I think this is a regression from HADOOP-9817. I modified some code and the 
 unit test passed. (See the attached patch.) However, I think the impact is 
 larger. I will add some unit tests to verify the behavior, and work on a more 
 complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9865:
--

Status: Patch Available  (was: Open)

 FileContext.globStatus() has a regression with respect to relative path
 ---

 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch, 
 HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch


 I discovered the problem when running unit test TestMRJobClient on Windows. 
 The cause is indirect in this case. In the unit test, we try to launch a job 
 and list its status. The job failed, and caused the list command get a result 
 of 0, which triggered the unit test assert. From the log and debug, the job 
 failed because we failed to create the Jar with classpath (see code around 
 {{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a 
 Windows specific step right now; so the test still passes on Linux. This step 
 failed because we passed in a relative path to {{FileContext.globStatus()}} 
 in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the 
 following.
 {noformat}
 2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] 
 launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch 
 container.
 org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
   at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
   at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
   at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
   at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
   at 
 org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
   at 
 org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {noformat}
 I think this is a regression from HADOOP-9817. I modified some code and the 
 unit test passed. (See the attached patch.) However, I think the impact is 
 larger. I will add some unit tests to verify the behavior, and work on a more 
 complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9865:
--

Affects Version/s: 2.3.0
   3.0.0

 FileContext.globStatus() has a regression with respect to relative path
 ---

 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-9865-demo.patch


 I discovered the problem when running unit test TestMRJobClient on Windows. 
 The cause is indirect in this case. In the unit test, we try to launch a job 
 and list its status. The job failed, and caused the list command get a result 
 of 0, which triggered the unit test assert. From the log and debug, the job 
 failed because we failed to create the Jar with classpath (see code around 
 {{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a 
 Windows specific step right now; so the test still passes on Linux. This step 
 failed because we passed in a relative path to {{FileContext.globStatus()}} 
 in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the 
 following.
 {noformat}
 2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] 
 launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch 
 container.
 org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
   at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
   at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
   at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
   at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
   at 
 org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
   at 
 org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {noformat}
 I think this is a regression from HADOOP-9817. I modified some code and the 
 unit test passed. (See the attached patch.) However, I think the impact is 
 larger. I will add some unit tests to verify the behavior, and work on a more 
 complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738807#comment-13738807
 ] 

Chuan Liu commented on HADOOP-9865:
---

Thanks for the suggestion, Colin! I was also thinking something similar. The 
patch was just to demo the problem in the description. I will add some unit 
tests to cover the scenarios as well!

 FileContext.globStatus() has a regression with respect to relative path
 ---

 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-9865-demo.patch


 I discovered the problem when running unit test TestMRJobClient on Windows. 
 The cause is indirect in this case. In the unit test, we try to launch a job 
 and list its status. The job failed, and caused the list command get a result 
 of 0, which triggered the unit test assert. From the log and debug, the job 
 failed because we failed to create the Jar with classpath (see code around 
 {{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a 
 Windows specific step right now; so the test still passes on Linux. This step 
 failed because we passed in a relative path to {{FileContext.globStatus()}} 
 in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the 
 following.
 {noformat}
 2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] 
 launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch 
 container.
 org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
   at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
   at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
   at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
   at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
   at 
 org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
   at 
 org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {noformat}
 I think this is a regression from HADOOP-9817. I modified some code and the 
 unit test passed. (See the attached patch.) However, I think the impact is 
 larger. I will add some unit tests to verify the behavior, and work on a more 
 complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-9865:
-

 Summary: FileContext.globStatus() has a regression with respect to 
relative path
 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu


I discovered the problem when running unit test TestMRJobClient. The cause is 
indirect in this case. In the unit test, we try to launch a job and list its 
status. The job failed, and caused the list command get a result of 0, which 
triggered the unit test assert. From the log and debug, the job failed because 
we failed to create the Jar with classpath (see code around 
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This step failed 
because we passed in a relative path to {{FileContext.globStatus()}} in 
{{FileUtil.createJarWithClassPath}}. The relevant log looks like the following.

{noformat}
2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] launcher.ContainerLaunch 
(ContainerLaunch.java:call(270)) - Failed to launch container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at 
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at 
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}

I think this is a regression from HADOOP-9817. I modified some code and the 
unit test passed. (See the attached patch.) However, I think the impact is 
larger. I will add some unit tests to verify the behavior, and work on a more 
complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9865:
--

Attachment: HADOOP-9865-demo.patch

Attach a patch demonstrating the problem described above. I will work on a more 
complete fix and add some unit tests.

 FileContext.globStatus() has a regression with respect to relative path
 ---

 Key: HADOOP-9865
 URL: https://issues.apache.org/jira/browse/HADOOP-9865
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: HADOOP-9865-demo.patch


 I discovered the problem when running unit test TestMRJobClient. The cause is 
 indirect in this case. In the unit test, we try to launch a job and list its 
 status. The job failed, and caused the list command get a result of 0, which 
 triggered the unit test assert. From the log and debug, the job failed 
 because we failed to create the Jar with classpath (see code around 
 {{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This step failed 
 because we passed in a relative path to {{FileContext.globStatus()}} in 
 {{FileUtil.createJarWithClassPath}}. The relevant log looks like the 
 following.
 {noformat}
 2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] 
 launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch 
 container.
 org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
   at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
   at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
   at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
   at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
   at 
 org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
   at 
 org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {noformat}
 I think this is a regression from HADOOP-9817. I modified some code and the 
 unit test passed. (See the attached patch.) However, I think the impact is 
 larger. I will add some unit tests to verify the behavior, and work on a more 
 complete fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9865:
--

Description: 
I discovered the problem when running unit test TestMRJobClient on Windows. The 
cause is indirect in this case. In the unit test, we try to launch a job and 
list its status. The job failed, and caused the list command get a result of 0, 
which triggered the unit test assert. From the log and debug, the job failed 
because we failed to create the Jar with classpath (see code around 
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a Windows 
specific step right now; so the test still passes on Linux. This step failed 
because we passed in a relative path to {{FileContext.globStatus()}} in 
{{FileUtil.createJarWithClassPath}}. The relevant log looks like the following.

{noformat}
2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] launcher.ContainerLaunch 
(ContainerLaunch.java:call(270)) - Failed to launch container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at 
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at 
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}

I think this is a regression from HADOOP-9817. I modified some code and the 
unit test passed. (See the attached patch.) However, I think the impact is 
larger. I will add some unit tests to verify the behavior, and work on a more 
complete fix.

  was:
I discovered the problem when running unit test TestMRJobClient. The cause is 
indirect in this case. In the unit test, we try to launch a job and list its 
status. The job failed, and caused the list command get a result of 0, which 
triggered the unit test assert. From the log and debug, the job failed because 
we failed to create the Jar with classpath (see code around 
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This step failed 
because we passed in a relative path to {{FileContext.globStatus()}} in 
{{FileUtil.createJarWithClassPath}}. The relevant log looks like the following.

{noformat}
2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] launcher.ContainerLaunch 
(ContainerLaunch.java:call(270)) - Failed to launch container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at 
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at 
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}

I think this is a regression from HADOOP-9817. I modified some code and the 
unit test passed. (See the attached patch.) However, I think the impact is 
larger. I will add some unit tests to verify the behavior, and work on a more 
complete fix.


 FileContext.globStatus() has a regression with respect to relative 

[jira] [Commented] (HADOOP-9802) Support Snappy codec on Windows.

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735804#comment-13735804
 ] 

Chuan Liu commented on HADOOP-9802:
---

+1

Sound good. If we run into scenarios HADOOP-8907 want to address, we can handle 
it in a separate JIRA.

 Support Snappy codec on Windows.
 

 Key: HADOOP-9802
 URL: https://issues.apache.org/jira/browse/HADOOP-9802
 Project: Hadoop Common
  Issue Type: Improvement
  Components: io
Affects Versions: 3.0.0, 1-win, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HADOOP-9802-branch-1-win.1.patch, 
 HADOOP-9802-trunk.1.patch, HADOOP-9802-trunk.2.patch, 
 HADOOP-9802-trunk.3.patch


 Build and test the existing Snappy codec on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9802) Support Snappy codec on Windows.

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735501#comment-13735501
 ] 

Chuan Liu commented on HADOOP-9802:
---

[~cnauroth], the change looks good!

In HADOOP-8907, Ivan added a convenient feature to load zlib1.dll from the same 
directory of native library dll.
I think it may be good to add the same feature for snappy for parity (at least 
for branch-1-win).
What do you think?

 Support Snappy codec on Windows.
 

 Key: HADOOP-9802
 URL: https://issues.apache.org/jira/browse/HADOOP-9802
 Project: Hadoop Common
  Issue Type: Improvement
  Components: io
Affects Versions: 3.0.0, 1-win, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HADOOP-9802-branch-1-win.1.patch, 
 HADOOP-9802-trunk.1.patch, HADOOP-9802-trunk.2.patch, 
 HADOOP-9802-trunk.3.patch


 Build and test the existing Snappy codec on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9831) Make checknative shell command accessible on Windows.

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733838#comment-13733838
 ] 

Chuan Liu commented on HADOOP-9831:
---

+1

The patch also works for me!

 Make checknative shell command accessible on Windows.
 -

 Key: HADOOP-9831
 URL: https://issues.apache.org/jira/browse/HADOOP-9831
 Project: Hadoop Common
  Issue Type: Improvement
  Components: bin
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
 Attachments: HADOOP-9831.1.patch


 The checknative command was implemented in HADOOP-9162 and HADOOP-9164 to 
 print information about availability of native libraries.  We already have 
 the native code to do this on Windows.  We just need to update hadoop.cmd to 
 expose the checknative command and pass through to the correct command class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9790) Job token path is not unquoted properly

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13732749#comment-13732749
 ] 

Chuan Liu commented on HADOOP-9790:
---

+1

bq. Oozie doesn't need to directly read HADOOP_TOKEN_FILE_LOCATION nor set it 
in the job conf. It's implicitly read by the UGI. The env is an implementation 
detail of how the container executor transfers tokens into a new process.

I think Ozzie uses DistCp in its mapreduce child tasks. Due to how we pass 
environment variable to child tasks on Windows, all code used by child tasks 
that accesses the environment variable values need to remove the surrounding 
quotes first.

 Job token path is not unquoted properly
 ---

 Key: HADOOP-9790
 URL: https://issues.apache.org/jira/browse/HADOOP-9790
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
 Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
 Attachments: HADOOP-9790.1.patch, HADOOP-9790.2.patch, stderr.txt


 Found during oozie unit tests (TestDistCpActionExecutor) and oozie ad-hoc 
 testing (distcp action).
 The error is:
 Exception reading 
 file:/D:/git/Monarch/project/oozie-monarch/core/target/test-data/minicluster/mapred/local/0_0/taskTracker/test/jobcache/job_20130725105336682_0001/jobToken.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9551) Backport common utils introduced with HADOOP-9413 to branch-1-win

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729940#comment-13729940
 ] 

Chuan Liu commented on HADOOP-9551:
---

Sorry. I think I mis-read your patch.

Should we fall back to Chris's original implementation instead of Java's API?
Because Java API's behavior seems incorrect according to the following bug 
report.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6203387

 Backport common utils introduced with HADOOP-9413 to branch-1-win
 -

 Key: HADOOP-9551
 URL: https://issues.apache.org/jira/browse/HADOOP-9551
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Attachments: HADOOP-9551.branch-1-win.common.2.patch, 
 HADOOP-9551.branch-1-win.common.3.patch, 
 HADOOP-9551.branch-1-win.common.4.patch


 Branch-1-win has the same set of problems described in HADOOP-9413. With this 
 Jira I plan to prepare a branch-1-win compatible patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9551) Backport common utils introduced with HADOOP-9413 to branch-1-win

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730062#comment-13730062
 ] 

Chuan Liu commented on HADOOP-9551:
---

bq. In practice, I think Windows will never execute the fallback. Since 
hadoop.dll is required for correctness on Windows in various ways, any 
realistic deployment will have the native code available.

Sound good to me! I also noticed trunk has the same logic. I already added +1 
before.

 Backport common utils introduced with HADOOP-9413 to branch-1-win
 -

 Key: HADOOP-9551
 URL: https://issues.apache.org/jira/browse/HADOOP-9551
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Attachments: HADOOP-9551.branch-1-win.common.2.patch, 
 HADOOP-9551.branch-1-win.common.3.patch, 
 HADOOP-9551.branch-1-win.common.4.patch


 Branch-1-win has the same set of problems described in HADOOP-9413. With this 
 Jira I plan to prepare a branch-1-win compatible patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9551) Backport common utils introduced with HADOOP-9413 to branch-1-win

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13727054#comment-13727054
 ] 

Chuan Liu commented on HADOOP-9551:
---

+1

I am not sure how do we resolve the native library dependency for downstream 
projects?
My observation in Hive seems like this: HADOOP_HOME environment variable needs 
to be set on Windows in order to run Hive unit tests because Hadoop needs to 
find 'winutils.exe'.
Once this change is in, native library will be required.
Hive may need to change its 'hadoop.cmd' to add 
'-Djava.library.path=%HADOOP_HOME%\lib\native\Windows_NT-amd64-64'.



 Backport common utils introduced with HADOOP-9413 to branch-1-win
 -

 Key: HADOOP-9551
 URL: https://issues.apache.org/jira/browse/HADOOP-9551
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Attachments: HADOOP-9551.branch-1-win.common.2.patch, 
 HADOOP-9551.branch-1-win.common.3.patch, 
 HADOOP-9551.branch-1-win.common.4.patch


 Branch-1-win has the same set of problems described in HADOOP-9413. With this 
 Jira I plan to prepare a branch-1-win compatible patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9801) Configuration#writeXml uses platform defaulting encoding, which may mishandle multi-byte characters.

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725835#comment-13725835
 ] 

Chuan Liu commented on HADOOP-9801:
---

+1 

Change looks good to me! Thanks, Chris!

 Configuration#writeXml uses platform defaulting encoding, which may mishandle 
 multi-byte characters.
 

 Key: HADOOP-9801
 URL: https://issues.apache.org/jira/browse/HADOOP-9801
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HADOOP-9801-branch-1.1.patch, HADOOP-9801-trunk.1.patch


 The overload of {{Configuration#writeXml}} that accepts an {{OutputStream}} 
 does not set encoding explicitly, so it chooses the platform default 
 encoding.  Depending on the platform's default encoding, this can cause 
 incorrect output data when encoding multi-byte characters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9776) HarFileSystem.listStatus() returns har://scheme-localhost:/... if port number is empty

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720841#comment-13720841
 ] 

Chuan Liu commented on HADOOP-9776:
---

The change looks good to me. Probably worth adding a unit test case to Hadoop 
to cover the scenario.

 HarFileSystem.listStatus() returns har://scheme-localhost:/... if port 
 number is empty
 --

 Key: HADOOP-9776
 URL: https://issues.apache.org/jira/browse/HADOOP-9776
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: shanyu zhao
 Attachments: HADOOP-9776.patch


 If the given har URI is har://scheme-localhost/usr/my.har/a, the result 
 of HarFileSystem.listStatus() will have a : appended after localhost, like 
 this: har://scheme-localhost:/usr/my.har/a. it should return 
 har://scheme-localhost/usr/my.bar/a instead.
 This creates problem when running a hive unit test TestCliDriver 
 (archive_excludeHadoop20.q), generating the following error:
   java.io.IOException: cannot find dir = 
 har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0
  in pathToPartitionInfo: 
 [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11,
  
 har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har]
   [junit] at 
 org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298)
   [junit] at 
 org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260)
   [junit] at 
 org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.init(CombineHiveInputFormat.java:104)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9774) RawLocalFileSystem.listStatus() return absolution paths when input path is relative on Windows

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720849#comment-13720849
 ] 

Chuan Liu commented on HADOOP-9774:
---

I think the behavior of {{RawLocalFileSystem.listStatus()}} will diverge with 
this patch, which is not a very good practice. If the intended behavior is the 
following, we should make it consistent on Linux and Windows in my opinion.

bq. When the caller use a relative path (without drive spec) to do listStatus() 
the resulting path should be relative.

It will be good to add a unit test case to cover this scenario as well.

 RawLocalFileSystem.listStatus() return absolution paths when input path is 
 relative on Windows
 --

 Key: HADOOP-9774
 URL: https://issues.apache.org/jira/browse/HADOOP-9774
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: shanyu zhao
 Attachments: HADOOP-9774.patch


 On Windows, when using RawLocalFileSystem.listStatus() to enumerate a 
 relative path (without drive spec), e.g., file:///mydata, the resulting 
 paths become absolute paths, e.g., [file://E:/mydata/t1.txt, 
 file://E:/mydata/t2.txt...].
 Note that if we use it to enumerate an absolute path, e.g., 
 file://E:/mydata then the we get the same results as above.
 This breaks some hive unit tests which uses local file system to simulate 
 HDFS when testing, therefore the drive spec is removed. Then after 
 listStatus() the path is changed to absolute path, hive failed to find the 
 path in its map reduce job.
 You'll see the following exception:
 [junit] java.io.IOException: cannot find dir = 
 pfile:/E:/GitHub/hive-monarch/build/ql/test/data/warehouse/src/kv1.txt in 
 pathToPartitionInfo: 
 [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/src]
 [junit]   at 
 org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298)
 This problem is introduced by this JIRA:
 HADOOP-8962
 Prior to the fix for HADOOP-8962 (merged in 0.23.5), the resulting paths are 
 relative paths if the parent paths are relative, e.g., 
 [file:///mydata/t1.txt, file:///mydata/t2.txt...]
 This behavior change is a side effect of the fix in HADOOP-8962, not an 
 intended change. The resulting behavior, even though is legitimate from a 
 function point of view, break consistency from the caller's point of view. 
 When the caller use a relative path (without drive spec) to do listStatus() 
 the resulting path should be relative. Therefore, I think this should be 
 fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9776) HarFileSystem.listStatus() returns har://scheme-localhost:/... if port number is empty

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721399#comment-13721399
 ] 

Chuan Liu commented on HADOOP-9776:
---

+1

nitpick: indentation seems wrong in the beginning of the test case.

 HarFileSystem.listStatus() returns har://scheme-localhost:/... if port 
 number is empty
 --

 Key: HADOOP-9776
 URL: https://issues.apache.org/jira/browse/HADOOP-9776
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: shanyu zhao
 Attachments: HADOOP-9776-2.patch, HADOOP-9776.patch


 If the given har URI is har://scheme-localhost/usr/my.har/a, the result 
 of HarFileSystem.listStatus() will have a : appended after localhost, like 
 this: har://scheme-localhost:/usr/my.har/a. it should return 
 har://scheme-localhost/usr/my.bar/a instead.
 This creates problem when running a hive unit test TestCliDriver 
 (archive_excludeHadoop20.q), generating the following error:
   java.io.IOException: cannot find dir = 
 har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0
  in pathToPartitionInfo: 
 [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11,
  
 har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har]
   [junit] at 
 org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298)
   [junit] at 
 org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260)
   [junit] at 
 org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.init(CombineHiveInputFormat.java:104)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719972#comment-13719972
 ] 

Chuan Liu commented on HADOOP-9507:
---

+1

The new changes look good to me!

 LocalFileSystem rename() is broken in some cases when destination exists
 

 Key: HADOOP-9507
 URL: https://issues.apache.org/jira/browse/HADOOP-9507
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
 Attachments: HADOOP-9507-branch-1.1.patch, 
 HADOOP-9507-branch-1.3.patch, HADOOP-9507-branch-1-win.1.patch, 
 HADOOP-9507-branch-1-win.3.patch, HADOOP-9507.branch-1-win.patch, 
 HADOOP-9507-trunk.1.patch, HADOOP-9507-trunk.2.patch, 
 HADOOP-9507-trunk.3.patch


 The rename() method in RawLocalFileSystem uses FileUtil.copy() without 
 realizing that FileUtil.copy() has a special behavior that if you're copying 
 /foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar 
 instead of overwriting it, which is not what rename() wants. So you end up 
 with weird behaviors like in this repro:
 {code}
 c:
 cd \
 md Foo
 md Bar
 md Foo\X
 md Bar\X
 hadoop fs -mv file:///c:/Foo file:///c:/Bar
 {code}
 At the end of this, you would expect to find only Bar\X, but you instead find 
 Bar\X\X.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718822#comment-13718822
 ] 

Chuan Liu commented on HADOOP-9507:
---

+1

Thanks for the detailed analysis, Chris!
I don't remember the subtle differences of File#renameTo() on Windows and Linux.
I verified the differences in renaming behavior, and agree with your fix.

 LocalFileSystem rename() is broken in some cases when destination exists
 

 Key: HADOOP-9507
 URL: https://issues.apache.org/jira/browse/HADOOP-9507
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
 Attachments: HADOOP-9507-branch-1.1.patch, 
 HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch, 
 HADOOP-9507-trunk.1.patch


 The rename() method in RawLocalFileSystem uses FileUtil.copy() without 
 realizing that FileUtil.copy() has a special behavior that if you're copying 
 /foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar 
 instead of overwriting it, which is not what rename() wants. So you end up 
 with weird behaviors like in this repro:
 {code}
 c:
 cd \
 md Foo
 md Bar
 md Foo\X
 md Bar\X
 hadoop fs -mv file:///c:/Foo file:///c:/Bar
 {code}
 At the end of this, you would expect to find only Bar\X, but you instead find 
 Bar\X\X.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719255#comment-13719255
 ] 

Chuan Liu commented on HADOOP-9768:
---

+1

Look good to me! Thanks for fix the problem, Chris!

 chown and chgrp reject users and groups with spaces on platforms where spaces 
 are otherwise acceptable
 --

 Key: HADOOP-9768
 URL: https://issues.apache.org/jira/browse/HADOOP-9768
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HADOOP-9768-branch-1.1.patch, 
 HADOOP-9768-branch-1.2.patch, HADOOP-9768-trunk.1.patch, 
 HADOOP-9768-trunk.2.patch


 The chown and chgrp commands enforce a check on a valid set of characters for 
 user and group.  The set of valid characters does not include space.  On some 
 platforms (notably Windows), the space character is acceptable.  We've seen 
 this cause test failures in {{TestFsShellReturnCode}} when running on Windows 
 if the logged-in user is a member of a group like Remote Desktop Users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9609) Remove sh dependency of bin-package target

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9609:
--

Attachment: HADOOP-9609-branch-1-win-3.patch

bq. You will copy hadoop libraries to $DIST_LIB_DIR/platform, while the 
original script copied them to $DIST_LIB_DIR when on Linux. right?

Nice catch! Attaching a new patch.

 Remove sh dependency of bin-package target
 --

 Key: HADOOP-9609
 URL: https://issues.apache.org/jira/browse/HADOOP-9609
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9609-branch-1-win-2.patch, 
 HADOOP-9609-branch-1-win-3.patch, HADOOP-9609-branch-1-win.patch


 In Ant package target, we no longer use packageBinNativeHadoop.sh to place 
 native library binaries. However, the same script is still present in the 
 bin-package target. We should remove bin-package target's dependency on sh to 
 keep it consistent with package target.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9759) Add support for NativeCodeLoader#getLibraryName on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9759:
--

Description: HADOOP-9164 introduced a new native API to get the library 
path for the Hadoop native libraries and other third party native libraries. 
There is no equivalent support of this native API on Windows in the original 
patch. This JIRA is created to track the work to support 
{{NativeCodeLoader#getLibraryName}}. We also have a build break on Windows due 
to the change.  (was: HADOOP-9164 introduced a new native API to get the 
library path for the Hadoop native libraries and other third party native 
libraries. There is no equivalent support of this native API in the original 
patch. This JIRA is created to track the work to support 
{{NativeCodeLoader#getLibraryName}}. We also have a build break on Windows due 
to the change.)

 Add support for NativeCodeLoader#getLibraryName on Windows
 --

 Key: HADOOP-9759
 URL: https://issues.apache.org/jira/browse/HADOOP-9759
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Critical
 Attachments: HADOOP-9759.patch


 HADOOP-9164 introduced a new native API to get the library path for the 
 Hadoop native libraries and other third party native libraries. There is no 
 equivalent support of this native API on Windows in the original patch. This 
 JIRA is created to track the work to support 
 {{NativeCodeLoader#getLibraryName}}. We also have a build break on Windows 
 due to the change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9755) HADOOP-9164 breaks the windows native build

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716599#comment-13716599
 ] 

Chuan Liu commented on HADOOP-9755:
---

Thanks for addressing the build break!
In HADOOP-9759, we provided a Windows version of 
{{NativeCodeLoader#getLibraryName}} instead of just skipping this method.

 HADOOP-9164 breaks the windows native build
 ---

 Key: HADOOP-9755
 URL: https://issues.apache.org/jira/browse/HADOOP-9755
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Vinay
Assignee: Binglin Chang
Priority: Blocker
 Attachments: HADOOP-9755.patch, HADOOP-9755.v2.patch


 After HADOOP-9164 hadooop windows native build is broken.
 {noformat}  NativeCodeLoader.c
 src\org\apache\hadoop\util\NativeCodeLoader.c(41): error C2065: 'Dl_info' : 
 undeclared identifier 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
 src\org\apache\hadoop\util\NativeCodeLoader.c(41): error C2146: syntax error 
 : missing ';' before identifier 'dl_info' 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
 src\org\apache\hadoop\util\NativeCodeLoader.c(41): error C2065: 'dl_info' : 
 undeclared identifier 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
 src\org\apache\hadoop\util\NativeCodeLoader.c(42): error C2143: syntax error 
 : missing ';' before 'type' 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
 src\org\apache\hadoop\util\NativeCodeLoader.c(45): error C2065: 'ret' : 
 undeclared identifier 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
 src\org\apache\hadoop\util\NativeCodeLoader.c(45): error C2065: 'dl_info' : 
 undeclared identifier 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
 src\org\apache\hadoop\util\NativeCodeLoader.c(45): error C2224: left of 
 '.dli_fname' must have struct/union type 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
 src\org\apache\hadoop\util\NativeCodeLoader.c(45): fatal error C1903: unable 
 to recover from previous error(s); stopping compilation 
 [D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
   NativeCrc32.c
 Done Building Project 
 D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj 
 (default targets) -- FAILED.
 Done Building Project 
 D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.sln 
 (default targets) -- FAILED.{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9759) Add support for NativeCodeLoader#getLibraryName on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9759:
--

Attachment: HADOOP-9759.2.patch

Attaching a new version with some slight update.

 Add support for NativeCodeLoader#getLibraryName on Windows
 --

 Key: HADOOP-9759
 URL: https://issues.apache.org/jira/browse/HADOOP-9759
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Critical
 Attachments: HADOOP-9759.2.patch, HADOOP-9759.patch


 HADOOP-9164 introduced a new native API to get the library path for the 
 Hadoop native libraries and other third party native libraries. There is no 
 equivalent support of this native API on Windows in the original patch. This 
 JIRA is created to track the work to support 
 {{NativeCodeLoader#getLibraryName}}. We also have a build break on Windows 
 due to the change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9759) Add support for NativeCodeLoader#getLibraryName on Windows

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-9759:
-

 Summary: Add support for NativeCodeLoader#getLibraryName on Windows
 Key: HADOOP-9759
 URL: https://issues.apache.org/jira/browse/HADOOP-9759
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Critical


HADOOP-9164 introduced a new native API to get the library path for the Hadoop 
native libraries and other third party native libraries. There is no equivalent 
support of this native API in the original patch. This JIRA is created to track 
the work to support {{NativeCodeLoader#getLibraryName}}.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9759) Add support for NativeCodeLoader#getLibraryName on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9759:
--

Description: HADOOP-9164 introduced a new native API to get the library 
path for the Hadoop native libraries and other third party native libraries. 
There is no equivalent support of this native API in the original patch. This 
JIRA is created to track the work to support 
{{NativeCodeLoader#getLibraryName}}. We also have a build break on Windows due 
to the change.  (was: HADOOP-9164 introduced a new native API to get the 
library path for the Hadoop native libraries and other third party native 
libraries. There is no equivalent support of this native API in the original 
patch. This JIRA is created to track the work to support 
{{NativeCodeLoader#getLibraryName}}.)

 Add support for NativeCodeLoader#getLibraryName on Windows
 --

 Key: HADOOP-9759
 URL: https://issues.apache.org/jira/browse/HADOOP-9759
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Critical

 HADOOP-9164 introduced a new native API to get the library path for the 
 Hadoop native libraries and other third party native libraries. There is no 
 equivalent support of this native API in the original patch. This JIRA is 
 created to track the work to support {{NativeCodeLoader#getLibraryName}}. We 
 also have a build break on Windows due to the change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9759) Add support for NativeCodeLoader#getLibraryName on Windows

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9759:
--

Attachment: HADOOP-9759.patch

Attaching a patch. When running the unit test, the function returns correct 
path to hadoop.dll, i.e. 
hadoop-common-project\hadoop-common\target\bin\hadoop.dll. However the unit 
test does not pass fully on Windows due to the native zlib compressor not 
available on Windows. 

 Add support for NativeCodeLoader#getLibraryName on Windows
 --

 Key: HADOOP-9759
 URL: https://issues.apache.org/jira/browse/HADOOP-9759
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Critical
 Attachments: HADOOP-9759.patch


 HADOOP-9164 introduced a new native API to get the library path for the 
 Hadoop native libraries and other third party native libraries. There is no 
 equivalent support of this native API in the original patch. This JIRA is 
 created to track the work to support {{NativeCodeLoader#getLibraryName}}. We 
 also have a build break on Windows due to the change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9759) Add support for NativeCodeLoader#getLibraryName on Windows

[Chuan Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715804#comment-13715804
 ] 

Chuan Liu commented on HADOOP-9759:
---

The unit test in my previous comment refers to new test case in 
TestNativeCodeLoader added by HADOOP-9164. -Drequire.test.libhadoop=true is 
required to run the unit test; otherwise, it will be skipped.

 Add support for NativeCodeLoader#getLibraryName on Windows
 --

 Key: HADOOP-9759
 URL: https://issues.apache.org/jira/browse/HADOOP-9759
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Critical
 Attachments: HADOOP-9759.patch


 HADOOP-9164 introduced a new native API to get the library path for the 
 Hadoop native libraries and other third party native libraries. There is no 
 equivalent support of this native API in the original patch. This JIRA is 
 created to track the work to support {{NativeCodeLoader#getLibraryName}}. We 
 also have a build break on Windows due to the change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9681) FileUtil.unTarUsingJava() should close the InputStream upon finishing

[Chuan Liu (JIRA)]

Chuan Liu created HADOOP-9681:
-

 Summary: FileUtil.unTarUsingJava() should close the InputStream 
upon finishing
 Key: HADOOP-9681
 URL: https://issues.apache.org/jira/browse/HADOOP-9681
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor


In {{FileUtil.unTarUsingJava()}} method, we did not close input steams 
explicitly upon finish. This could lead to a file handle leak on Windows.

I discovered this when investigating the unit test case failure of 
{{TestFSDownload.testDownloadArchive()}}. FSDownload class will use 
{{FileUtil.unTarUsingJava()}} to unpack some temporary archive file. Later, the 
temporary file should be deleted. Because of the file handle leak, the 
{{File.delete()}} method fails. The test case then fails because it assert the 
temporary file should not exist.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9681) FileUtil.unTarUsingJava() should close the InputStream upon finishing

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9681:
--

Attachment: HADOOP-9681-trunk.patch

Attaching a patch that closes the input streams in the {{unTarUsingJava}} 
method.

 FileUtil.unTarUsingJava() should close the InputStream upon finishing
 -

 Key: HADOOP-9681
 URL: https://issues.apache.org/jira/browse/HADOOP-9681
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9681-trunk.patch


 In {{FileUtil.unTarUsingJava()}} method, we did not close input steams 
 explicitly upon finish. This could lead to a file handle leak on Windows.
 I discovered this when investigating the unit test case failure of 
 {{TestFSDownload.testDownloadArchive()}}. FSDownload class will use 
 {{FileUtil.unTarUsingJava()}} to unpack some temporary archive file. Later, 
 the temporary file should be deleted. Because of the file handle leak, the 
 {{File.delete()}} method fails. The test case then fails because it assert 
 the temporary file should not exist.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9681) FileUtil.unTarUsingJava() should close the InputStream upon finishing

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9681:
--

Attachment: HADOOP-9681-trunk.2.patch

Thanks for the suggestion, Chris! A new patch using {{IOUtils#cleanup}} is 
attached. I will prepare a branch-1-win patch shortly.

 FileUtil.unTarUsingJava() should close the InputStream upon finishing
 -

 Key: HADOOP-9681
 URL: https://issues.apache.org/jira/browse/HADOOP-9681
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9681-trunk.2.patch, HADOOP-9681-trunk.patch


 In {{FileUtil.unTarUsingJava()}} method, we did not close input steams 
 explicitly upon finish. This could lead to a file handle leak on Windows.
 I discovered this when investigating the unit test case failure of 
 {{TestFSDownload.testDownloadArchive()}}. FSDownload class will use 
 {{FileUtil.unTarUsingJava()}} to unpack some temporary archive file. Later, 
 the temporary file should be deleted. Because of the file handle leak, the 
 {{File.delete()}} method fails. The test case then fails because it assert 
 the temporary file should not exist.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9681) FileUtil.unTarUsingJava() should close the InputStream upon finishing

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9681:
--

Attachment: HADOOP-9681-branch-1-win.patch

Attach a branch-1-win patch.

 FileUtil.unTarUsingJava() should close the InputStream upon finishing
 -

 Key: HADOOP-9681
 URL: https://issues.apache.org/jira/browse/HADOOP-9681
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 1-win, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9681-branch-1-win.patch, 
 HADOOP-9681-trunk.2.patch, HADOOP-9681-trunk.patch


 In {{FileUtil.unTarUsingJava()}} method, we did not close input steams 
 explicitly upon finish. This could lead to a file handle leak on Windows.
 I discovered this when investigating the unit test case failure of 
 {{TestFSDownload.testDownloadArchive()}}. FSDownload class will use 
 {{FileUtil.unTarUsingJava()}} to unpack some temporary archive file. Later, 
 the temporary file should be deleted. Because of the file handle leak, the 
 {{File.delete()}} method fails. The test case then fails because it assert 
 the temporary file should not exist.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9681) FileUtil.unTarUsingJava() should close the InputStream upon finishing

[Chuan Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-9681:
--

Affects Version/s: 1-win

 FileUtil.unTarUsingJava() should close the InputStream upon finishing
 -

 Key: HADOOP-9681
 URL: https://issues.apache.org/jira/browse/HADOOP-9681
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0, 1-win, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Attachments: HADOOP-9681-branch-1-win.patch, 
 HADOOP-9681-trunk.2.patch, HADOOP-9681-trunk.patch


 In {{FileUtil.unTarUsingJava()}} method, we did not close input steams 
 explicitly upon finish. This could lead to a file handle leak on Windows.
 I discovered this when investigating the unit test case failure of 
 {{TestFSDownload.testDownloadArchive()}}. FSDownload class will use 
 {{FileUtil.unTarUsingJava()}} to unpack some temporary archive file. Later, 
 the temporary file should be deleted. Because of the file handle leak, the 
 {{File.delete()}} method fails. The test case then fails because it assert 
 the temporary file should not exist.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira