[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Attachment: HADOOP-11639.03.patch
o3.Patch is rebased to current trunk and also fixes the new PSAPI_VERSION macro
redefinition warning.
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch,
HADOOP-11639.02.patch, HADOOP-11639.03.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11691) X86 build of libwinutils is broken
[
https://issues.apache.org/jira/browse/HADOOP-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375872#comment-14375872
]
Remus Rusanu commented on HADOOP-11691:
---
Hi [~kiranmr] the trunk still fail for me to build x86:
{code}
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj
(default target) (4) -
(Link target) -
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
: fatal error LNK1112: module machine type 'x
64' conflicts with target machine type 'X86'
[E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
{code}
I think the
{code}
- TargetEnvironmentX64/TargetEnvironment
+ TargetEnvironment$(Platform)/TargetEnvironment
{code}
is incorrect because $(Platform) expands to `Release|Win32` or `Debug|Win32`
but the MIDL Task https://msdn.microsoft.com/en-us/library/ee862478.aspx
expects `win32` or `x64`.
X86 build of libwinutils is broken
--
Key: HADOOP-11691
URL: https://issues.apache.org/jira/browse/HADOOP-11691
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 3.0.0
Reporter: Remus Rusanu
Assignee: Kiran Kumar M R
Attachments: HADOOP-11691-001.patch, HADOOP-11691-002.patch,
HADOOP-11691-003.patch
Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results
in error:
{code}
(Link target) -
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
: fatal error LNK1112: module machine type 'x64' conflicts with target
machine type 'X86'
[E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11691) X86 build of libwinutils is broken
[
https://issues.apache.org/jira/browse/HADOOP-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375877#comment-14375877
]
Remus Rusanu commented on HADOOP-11691:
---
Actually $(Platform) is Win32, not sure what makes my midl unhappy. I'll
investigate.
X86 build of libwinutils is broken
--
Key: HADOOP-11691
URL: https://issues.apache.org/jira/browse/HADOOP-11691
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 3.0.0
Reporter: Remus Rusanu
Assignee: Kiran Kumar M R
Attachments: HADOOP-11691-001.patch, HADOOP-11691-002.patch,
HADOOP-11691-003.patch
Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results
in error:
{code}
(Link target) -
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
: fatal error LNK1112: module machine type 'x64' conflicts with target
machine type 'X86'
[E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11691) X86 build of libwinutils is broken
[
https://issues.apache.org/jira/browse/HADOOP-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375902#comment-14375902
]
Remus Rusanu commented on HADOOP-11691:
---
Can you please tell me the midl command line you get? For me is this one:
{code}
E:\HW\tools\OACR\bin\midl.exe /W2 /WX /nologo /char signed /env win32 /Oicf
/app_config /outE:\HW\project\hadoop-common\hadoop-common-project\hadoo
p-common\target/winutils/ /h hadoopwinutilsvc_h.h /tlb
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/libwinutils
.tlb /robust hadoopwinutilsvc.idl
{code}
which passes in `/env win32` as per the MIDL MSbuild Task spec, but apparently
midl.exe wants `/win32` instead. I'm courious in your case what does the midl
command line look like.
X86 build of libwinutils is broken
--
Key: HADOOP-11691
URL: https://issues.apache.org/jira/browse/HADOOP-11691
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 3.0.0
Reporter: Remus Rusanu
Assignee: Kiran Kumar M R
Attachments: HADOOP-11691-001.patch, HADOOP-11691-002.patch,
HADOOP-11691-003.patch
Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results
in error:
{code}
(Link target) -
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
: fatal error LNK1112: module machine type 'x64' conflicts with target
machine type 'X86'
[E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11691) X86 build of libwinutils is broken
[
https://issues.apache.org/jira/browse/HADOOP-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363021#comment-14363021
]
Remus Rusanu commented on HADOOP-11691:
---
Ideally {code}C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\midl.exe{code}
would be consistent with {code}Processing C:\Program Files\Windows
Kits\8.1\Include\um\oaidl.idl{code}.
Overall I'm not to worried, the idl part is only relevant for the secure
container executor and IDLs compiled by SDK 7.1 and SDK 8.1 must be compatible.
X86 build of libwinutils is broken
--
Key: HADOOP-11691
URL: https://issues.apache.org/jira/browse/HADOOP-11691
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 3.0.0
Reporter: Remus Rusanu
Assignee: Kiran Kumar M R
Attachments: HADOOP-11691-001.patch, HADOOP-11691-002.patch
Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results
in error:
{code}
(Link target) -
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
: fatal error LNK1112: module machine type 'x64' conflicts with target
machine type 'X86'
[E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352818#comment-14352818
]
Remus Rusanu commented on HADOOP-11639:
---
I created HADOOP-11691 for the x86 build failure
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch,
HADOOP-11639.02.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352811#comment-14352811
]
Remus Rusanu commented on HADOOP-11639:
---
[~chuanliu] After YARN-2190 trunk no longer compiles x86:
{code}
(Link target) -
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
: fatal error LNK1112: module machine type 'x
64' conflicts with target machine type 'X86'
[E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
{code}
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch,
HADOOP-11639.02.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11691) X86 build of libwinutils is broken
Remus Rusanu created HADOOP-11691:
-
Summary: X86 build of libwinutils is broken
Key: HADOOP-11691
URL: https://issues.apache.org/jira/browse/HADOOP-11691
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 3.0.0
Reporter: Remus Rusanu
Hadoop-9922 recently fixed x86 build. After YARN-2190 compiling x86 results in
error:
{code}
(Link target) -
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\target/winutils/hadoopwinutilsvc_s.obj
: fatal error LNK1112: module machine type 'x64' conflicts with target machine
type 'X86'
[E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Attachment: HADOOP-11639.02.patch
02.patch builds clean for x86 target
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch,
HADOOP-11639.02.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Status: Patch Available (was: Open)
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch,
HADOOP-11639.02.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Status: Open (was: Patch Available)
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344970#comment-14344970
]
Remus Rusanu commented on HADOOP-11639:
---
Thanks [~kirankumarmr], I tested 32 bit build with .02.patch and builds clean.
I used
{code}
- int count = 0;
- int crt = 0;
+ size_t count = 0;
+ size_t crt = 0;
{code}
to have the same type for crt and count.
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch,
HADOOP-11639.02.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Attachment: HADOOP-11639.00.patch
00.Patch fixes all warnings in building the two libwinutils projects. The fixes
include:
- remove unreferenced variables
- fix several const correctness issues
- fix some of incorrect .h vs. .c function declarations
- fix some missing .h function declarations
- fix some incorrect signed/unsigned types (int. vs size_t vs. DWORD)
- fix some incorrectly declared buffers
- explicit cast when required (eg. all HANDLE handling(!) in .idl related APIs)
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Affects Version/s: 2.6.0
Status: Patch Available (was: Open)
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342979#comment-14342979
]
Remus Rusanu commented on HADOOP-11639:
---
Forgot to pull trunk...
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Status: Open (was: Patch Available)
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Attachment: HADOOP-11639.01.patch
Now with current trunk
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11639:
--
Status: Patch Available (was: Open)
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343084#comment-14343084
]
Remus Rusanu commented on HADOOP-11639:
---
My patch overwrites some changes from [~kirankumarmr] for HADOOP-9922. this is
simply because originally I did all the changes on an old trunk. After the pull
I resolved the merge by taking my changes on the conflict. Specifically, in
HADOOP-9922:
{code}
hadoop-common-project/hadoop-common/src/main/winutils/service.c
@@ -206,7 +206,7 @@ DWORD ValidateConfigurationFile() {
- int crt = 0, crtSid = 0;
+ unsigned int crt = 0, crtSid = 0;
{code}
while on my patch is:
{code}
hadoop-common-project/hadoop-common/src/main/winutils/service.c
@@ -206,7 +206,7 @@ DWORD ValidateConfigurationFile() {
- unsigned int crt = 0, crtSid = 0;
+ DWORD crt = 0;
{code}
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch, HADOOP-11639.01.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342964#comment-14342964
]
Remus Rusanu commented on HADOOP-11639:
---
I still have some C4748 warnings not sure how to handle these:
{code}
E:\HW\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj
(default target) (2) -
(Link target) -
e:\hw\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.c(533):
warning C4748: /GS can not protect parameters
and local variables from local buffer overrun because optimizations are
disabled in function [E:\HW\project\hadoop-common\hadoop-common-project\hadoop
-common\src\main\native\native.vcxproj]
e:\hw\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.c(2360):
warning C4748: /GS can not protect parameters
and local variables from local buffer overrun because optimizations are
disabled in function [E:\HW\project\hadoop-common\hadoop-common-project\hadoo
p-common\src\main\native\native.vcxproj]
e:\hw\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.c(2311):
warning C4748: /GS can not protect parameters
and local variables from local buffer overrun because optimizations are
disabled in function [E:\HW\project\hadoop-common\hadoop-common-project\hadoo
p-common\src\main\native\native.vcxproj]
e:\hw\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.c(342):
warning C4748: /GS can not protect parameters
and local variables from local buffer overrun because optimizations are
disabled in function [E:\HW\project\hadoop-common\hadoop-common-project\hadoop
-common\src\main\native\native.vcxproj]
e:\hw\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\client.c(46):
warning C4748: /GS can not protect parameters and lo
cal variables from local buffer overrun because optimizations are disabled in
function [E:\HW\project\hadoop-common\hadoop-common-project\hadoop-commo
n\src\main\native\native.vcxproj]
e:\hw\project\hadoop-common\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.c(815):
warning C4748: /GS can not protect parameters
and local variables from local buffer overrun because optimizations are
disabled in function [E:\HW\project\hadoop-common\hadoop-common-project\hadoop
-common\src\main\native\native.vcxproj]
{code}
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342967#comment-14342967
]
Remus Rusanu commented on HADOOP-11639:
---
BTW, I also fixed the pre YARN-2198 _wfopen warning
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Remus Rusanu
Attachments: HADOOP-11639.00.patch
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HADOOP-11639) Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
[
https://issues.apache.org/jira/browse/HADOOP-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu reassigned HADOOP-11639:
-
Assignee: Remus Rusanu
Clean up Windows native code compilation warnings related to Windows Secure
Container Executor.
---
Key: HADOOP-11639
URL: https://issues.apache.org/jira/browse/HADOOP-11639
Project: Hadoop Common
Issue Type: Bug
Components: native
Reporter: Chris Nauroth
Assignee: Remus Rusanu
YARN-2198 introduced additional code in Hadoop Common to support the
NodeManager {{WindowsSecureContainerExecutor}}. The patch introduced new
compilation warnings that we need to investigate and resolve.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11321) copyToLocal cannot save a file to an SMB share unless the user has Full Control permissions.
[
https://issues.apache.org/jira/browse/HADOOP-11321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240864#comment-14240864
]
Remus Rusanu commented on HADOOP-11321:
---
The WSCE has no versioning in the IDL. As the calls are always local, it is
expected that the binaries deployed are self-consistent.
copyToLocal cannot save a file to an SMB share unless the user has Full
Control permissions.
Key: HADOOP-11321
URL: https://issues.apache.org/jira/browse/HADOOP-11321
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-11321.003.patch, HADOOP-11321.004.patch,
HADOOP-11321.1.patch, HADOOP-11321.2.patch, winutils.tmp.patch
In Hadoop 2, it is impossible to use {{copyToLocal}} to copy a file from HDFS
to a destination on an SMB share. This is because in Hadoop 2, the
{{copyToLocal}} maps to 2 underlying {{RawLocalFileSystem}} operations:
{{create}} and {{setPermission}}. On an SMB share, the user may be
authorized for the {{create}} but denied for the {{setPermission}}. Windows
denies the {{WRITE_DAC}} right required by {{setPermission}} unless the user
has Full Control permissions. Granting Full Control isn't feasible for most
deployments, because it's insecure. This is a regression from Hadoop 1,
where {{copyToLocal}} only did a {{create}} and didn't do a separate
{{setPermission}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11321) copyToLocal cannot save a file to an SMB share unless the user has Full Control permissions.
[
https://issues.apache.org/jira/browse/HADOOP-11321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240866#comment-14240866
]
Remus Rusanu commented on HADOOP-11321:
---
The WSCE file operations should only apply to container launch/localization.
Even if all the NM operations all succeed on a SMB share, when the impersonated
container launched by WSCE will fail to launch on a SMB share because it
implies Kerberos delegation and the S4U token used by the WSCE impersonation
model does not support delegation.
copyToLocal cannot save a file to an SMB share unless the user has Full
Control permissions.
Key: HADOOP-11321
URL: https://issues.apache.org/jira/browse/HADOOP-11321
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.6.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-11321.003.patch, HADOOP-11321.004.patch,
HADOOP-11321.1.patch, HADOOP-11321.2.patch, winutils.tmp.patch
In Hadoop 2, it is impossible to use {{copyToLocal}} to copy a file from HDFS
to a destination on an SMB share. This is because in Hadoop 2, the
{{copyToLocal}} maps to 2 underlying {{RawLocalFileSystem}} operations:
{{create}} and {{setPermission}}. On an SMB share, the user may be
authorized for the {{create}} but denied for the {{setPermission}}. Windows
denies the {{WRITE_DAC}} right required by {{setPermission}} unless the user
has Full Control permissions. Granting Full Control isn't feasible for most
deployments, because it's insecure. This is a regression from Hadoop 1,
where {{copyToLocal}} only did a {{create}} and didn't do a separate
{{setPermission}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11080) Convert Windows native build in hadoop-common to use CMake.
[ https://issues.apache.org/jira/browse/HADOOP-11080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128324#comment-14128324 ] Remus Rusanu commented on HADOOP-11080: --- YARN-2198, YARN-1972 and YARN-2458 patches come with some big changes to the winutils/libwinutils build, including resource compile and midl step for IDL compile. My changes also cleaned up the msbuild, eg. use the maven target as intermediate build location and so on. As we already have CMake dependency for hdfs, I have no objections to swithing to it, but well have to validate the more complex build steps that are coming with the Windows secure nodemanager work. Convert Windows native build in hadoop-common to use CMake. --- Key: HADOOP-11080 URL: https://issues.apache.org/jira/browse/HADOOP-11080 Project: Hadoop Common Issue Type: Improvement Components: build, native Reporter: Chris Nauroth The Windows native build in hadoop-common currently relies on a set of checked-in MSBuild project files. The logic of these project files is largely redundant with the logic of CMakeLists.txt. It causes some dual maintenance overhead, because certain build changes require separate changes for both CMake and MSBuild. This issue proposes to convert the Windows build process to use CMake instead of checked-in project files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11003) org.apache.hadoop.util.Shell should not take a dependency on binaries being deployed when used as a library
Remus Rusanu created HADOOP-11003:
-
Summary: org.apache.hadoop.util.Shell should not take a dependency
on binaries being deployed when used as a library
Key: HADOOP-11003
URL: https://issues.apache.org/jira/browse/HADOOP-11003
Project: Hadoop Common
Issue Type: Bug
Components: util
Environment: Windows
Reporter: Remus Rusanu
HIVE-7845 shows how an exception is being thrown when
org.apache.hadoop.util.Shell is being used as a library, not as part of a
deployed Hadoop environment.
{code}
13:20:00 [ERROR pool-2-thread-4 Shell.getWinUtilsPath] Failed to locate the
winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at
org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:324)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:339)
at org.apache.hadoop.util.Shell.clinit(Shell.java:332)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:918)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.clinit(HiveConf.java:228)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-11003) org.apache.hadoop.util.Shell should not take a dependency on binaries being deployed when used as a library
[
https://issues.apache.org/jira/browse/HADOOP-11003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-11003:
--
Description:
HIVE-7845 shows how an exception is being thrown when
org.apache.hadoop.util.Shell is being used as a library, not as part of a
deployed Hadoop environment.
{code}
13:20:00 [ERROR pool-2-thread-4 Shell.getWinUtilsPath] Failed to locate the
winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at
org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:324)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:339)
at org.apache.hadoop.util.Shell.clinit(Shell.java:332)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:918)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.clinit(HiveConf.java:228)
{code}
There are similar native dependencies (eg. NativeIO and hadoop.dll) that handle
lack of binaries with fallback to non-native code paths.
was:
HIVE-7845 shows how an exception is being thrown when
org.apache.hadoop.util.Shell is being used as a library, not as part of a
deployed Hadoop environment.
{code}
13:20:00 [ERROR pool-2-thread-4 Shell.getWinUtilsPath] Failed to locate the
winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at
org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:324)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:339)
at org.apache.hadoop.util.Shell.clinit(Shell.java:332)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:918)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.clinit(HiveConf.java:228)
{code}
org.apache.hadoop.util.Shell should not take a dependency on binaries being
deployed when used as a library
---
Key: HADOOP-11003
URL: https://issues.apache.org/jira/browse/HADOOP-11003
Project: Hadoop Common
Issue Type: Bug
Components: util
Environment: Windows
Reporter: Remus Rusanu
HIVE-7845 shows how an exception is being thrown when
org.apache.hadoop.util.Shell is being used as a library, not as part of a
deployed Hadoop environment.
{code}
13:20:00 [ERROR pool-2-thread-4 Shell.getWinUtilsPath] Failed to locate the
winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at
org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:324)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:339)
at org.apache.hadoop.util.Shell.clinit(Shell.java:332)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:918)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.clinit(HiveConf.java:228)
{code}
There are similar native dependencies (eg. NativeIO and hadoop.dll) that
handle lack of binaries with fallback to non-native code paths.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029679#comment-14029679 ] Remus Rusanu commented on HADOOP-10663: --- [~cnauroth] I agree, lets go with MAPREDUCE-5912. But on the argument of the hypothetical FS that accepts colon I would say than, on the same logic, there will never ever be a way to determine the FS from a Path that has not scheme. Any heuristic that infers the FS from name can be defeated by a carefully constructed future FS :) In other words scheme should be mandatory Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Assignee: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HADOOP-10663: -- Status: Open (was: Patch Available) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Assignee: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HADOOP-10663: -- Attachment: HADOOP-10663.1.patch Reload patch again to trigger QA Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HADOOP-10663: -- Status: Open (was: Patch Available) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HADOOP-10663: -- Status: Patch Available (was: Open) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HADOOP-10663: -- Attachment: (was: HADOOP-10663.1.patch) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10664) TestNetUtils.testNormalizeHostName fails
[ https://issues.apache.org/jira/browse/HADOOP-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019622#comment-14019622 ] Remus Rusanu commented on HADOOP-10664: --- The DNS for 3w.org is back today: 213.175.197.40. [~andrew.wang] it has to be a name that starts with a number, that's what the test is validating (regression test for HADOOP-8372). Of course Apache could register such a subdomain and then we could use that. TestNetUtils.testNormalizeHostName fails Key: HADOOP-10664 URL: https://issues.apache.org/jira/browse/HADOOP-10664 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Chen He Labels: test java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.junit.Assert.assertFalse(Assert.java:74) at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:617) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HADOOP-10663: -- Affects Version/s: 3.0.0 Status: Patch Available (was: Open) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018737#comment-14018737 ] Remus Rusanu commented on HADOOP-10663: --- The test failure does not seem to be related to the patch. Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018774#comment-14018774 ] Remus Rusanu commented on HADOOP-10663: --- OK, the test fails because it has a dependency on 3w.org being resolved in DNS. That is 3w.org, not w3.org. 3w.org does no resolve today. So I guess the 'possible' issue in HADOOP-8372 has become a real issue... Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10664) TestNetUtils.testNormalizeHostName fails
[ https://issues.apache.org/jira/browse/HADOOP-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018782#comment-14018782 ] Remus Rusanu commented on HADOOP-10664: --- HADOOP-8372 introduced a dependency on 3w.org to resolve via DNS. Apparently the host no longer resolves today. TestNetUtils.testNormalizeHostName fails Key: HADOOP-10664 URL: https://issues.apache.org/jira/browse/HADOOP-10664 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Chen He java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.junit.Assert.assertFalse(Assert.java:74) at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:617) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018784#comment-14018784 ] Remus Rusanu commented on HADOOP-10663: --- HADOOP-10664 tracks the infra test failure Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10664) TestNetUtils.testNormalizeHostName fails
[ https://issues.apache.org/jira/browse/HADOOP-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018806#comment-14018806 ] Remus Rusanu commented on HADOOP-10664: --- The fix is a bit more complicated. For one, every branch affected must be fixed. But more important is to find a fix that does not suffer from the same issue (ie. not simply pick another host that could fail tomorrow). I for one would vote for the Apache foundation to register a DNS, eg. 1test.apache.org or similar, and then use that DNS name in the unit test. But such a solution is beyond my bandwith, and beyond my leverage with the foundation. So I would rather leave this for someone more experienced to pick up. I'm sure it will be a pretty hot issue soon if the 3w.org stays down, all patches uploaded will start fail... TestNetUtils.testNormalizeHostName fails Key: HADOOP-10664 URL: https://issues.apache.org/jira/browse/HADOOP-10664 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Chen He java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.junit.Assert.assertFalse(Assert.java:74) at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:617) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10664) TestNetUtils.testNormalizeHostName fails
[
https://issues.apache.org/jira/browse/HADOOP-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018814#comment-14018814
]
Remus Rusanu commented on HADOOP-10664:
---
FYI, 3w.org was updated recently:
{code}
whois 3w.org
...
Updated Date: 2014-06-04T14:23:31Z
{code}
TestNetUtils.testNormalizeHostName fails
Key: HADOOP-10664
URL: https://issues.apache.org/jira/browse/HADOOP-10664
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chen He
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:617)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10659) Refactor AccessControlList to reuse utility functions and to improve performance
[ https://issues.apache.org/jira/browse/HADOOP-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018819#comment-14018819 ] Remus Rusanu commented on HADOOP-10659: --- Sry, I posted ont he wrong JIRA. Nevermind my comment. Refactor AccessControlList to reuse utility functions and to improve performance Key: HADOOP-10659 URL: https://issues.apache.org/jira/browse/HADOOP-10659 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: HADOOP-10659.patch Minor improvements can be done on _AccessControlList_. Code Reusability: _AccessControlList_ sanitizes the input list to remove duplicate entries, trim entries. _StringUtils.getTrimmedStringCollection_ can be used in this case. Performance: _AccessControlList_ uses TreeSet to maintain set of users and groups. HashSet improves the performance slightly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10659) Refactor AccessControlList to reuse utility functions and to improve performance
[ https://issues.apache.org/jira/browse/HADOOP-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018815#comment-14018815 ] Remus Rusanu commented on HADOOP-10659: --- Test failure is HADOOP-10664 Refactor AccessControlList to reuse utility functions and to improve performance Key: HADOOP-10659 URL: https://issues.apache.org/jira/browse/HADOOP-10659 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: HADOOP-10659.patch Minor improvements can be done on _AccessControlList_. Code Reusability: _AccessControlList_ sanitizes the input list to remove duplicate entries, trim entries. _StringUtils.getTrimmedStringCollection_ can be used in this case. Performance: _AccessControlList_ uses TreeSet to maintain set of users and groups. HashSet improves the performance slightly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
Remus Rusanu created HADOOP-10663: - Summary: Path.getFileSystem should identify Windows drive (c:) like files and return the localFS Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Reporter: Remus Rusanu While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10644) Remote principal name case sensitivity issue introduced on Windows by HADOOP-10418
Remus Rusanu created HADOOP-10644:
-
Summary: Remote principal name case sensitivity issue introduced
on Windows by HADOOP-10418
Key: HADOOP-10644
URL: https://issues.apache.org/jira/browse/HADOOP-10644
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.4.0
Reporter: Remus Rusanu
HADOOP-10418 caused the SPN to be generated using KRB_NT_SRV_HST type. This
results in a wrong case FQDN name and the
{code}
isPrincipalValid = serverPrincipal.equals(confPrincipal);
{code}
check fails due to case difference.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10644) Remote principal name case sensitivity issue introduced on Windows by HADOOP-10418
[
https://issues.apache.org/jira/browse/HADOOP-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HADOOP-10644:
--
Labels: windows (was: )
Remote principal name case sensitivity issue introduced on Windows by
HADOOP-10418
--
Key: HADOOP-10644
URL: https://issues.apache.org/jira/browse/HADOOP-10644
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.4.0
Reporter: Remus Rusanu
Labels: windows
HADOOP-10418 caused the SPN to be generated using KRB_NT_SRV_HST type. This
results in a wrong case FQDN name and the
{code}
isPrincipalValid = serverPrincipal.equals(confPrincipal);
{code}
check fails due to case difference.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10644) Remote principal name case sensitivity issue introduced on Windows by HADOOP-10418
[
https://issues.apache.org/jira/browse/HADOOP-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013625#comment-14013625
]
Remus Rusanu commented on HADOOP-10644:
---
Note that as per RFC4120 the fix is to make the confPrincipal lowercase:
{code}
6.2.1. Name of Server Principals
... Where the name
of the host
is not case sensitive (for example, with Internet domain names) the
name of the host MUST be lowercase
{code}
Remote principal name case sensitivity issue introduced on Windows by
HADOOP-10418
--
Key: HADOOP-10644
URL: https://issues.apache.org/jira/browse/HADOOP-10644
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.4.0
Reporter: Remus Rusanu
Labels: windows
HADOOP-10418 caused the SPN to be generated using KRB_NT_SRV_HST type. This
results in a wrong case FQDN name and the
{code}
isPrincipalValid = serverPrincipal.equals(confPrincipal);
{code}
check fails due to case difference.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-10644) Remote principal name case sensitivity issue introduced on Windows by HADOOP-10418
[
https://issues.apache.org/jira/browse/HADOOP-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu resolved HADOOP-10644.
---
Resolution: Not a Problem
ok, the issue comes from config. the principals in *-site.zml must be properly
cased (lowercase hostnames). SecurityUtil.replacePatternt would do the right
thing, had it have a chance.
Remote principal name case sensitivity issue introduced on Windows by
HADOOP-10418
--
Key: HADOOP-10644
URL: https://issues.apache.org/jira/browse/HADOOP-10644
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.4.0
Reporter: Remus Rusanu
Labels: windows
HADOOP-10418 caused the SPN to be generated using KRB_NT_SRV_HST type. This
results in a wrong case FQDN name and the
{code}
isPrincipalValid = serverPrincipal.equals(confPrincipal);
{code}
check fails due to case difference.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
