[jira] [Created] (HADOOP-8879) TestUserGroupInformation fails on Windows when runas Administrator
Bikas Saha created HADOOP-8879: -- Summary: TestUserGroupInformation fails on Windows when runas Administrator Key: HADOOP-8879 URL: https://issues.apache.org/jira/browse/HADOOP-8879 Project: Hadoop Common Issue Type: Bug Reporter: Bikas Saha User name is case insensitive on Windows and whoami returns administrator instead of Administrator causing test assertion to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8847) Change untar to use Java API instead of spawning tar process
Bikas Saha created HADOOP-8847: -- Summary: Change untar to use Java API instead of spawning tar process Key: HADOOP-8847 URL: https://issues.apache.org/jira/browse/HADOOP-8847 Project: Hadoop Common Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha Currently FileUtil.unTar() spawns tar utility to do the work. Tar may not be present on all platforms by default eg. Windows. So changing this to use JAVA API's would help make it more cross-platform. FileUtil.unZip() uses the same approach. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8836) UGI should throw exception in case winutils.exe cannot be loaded
Bikas Saha created HADOOP-8836: -- Summary: UGI should throw exception in case winutils.exe cannot be loaded Key: HADOOP-8836 URL: https://issues.apache.org/jira/browse/HADOOP-8836 Project: Hadoop Common Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha In upstream projects like Hive its hard to see why getting user group information failed because the API swallows the exception. One of the cases is when winutils is not present where Hadoop expects it to be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8250) Investigate uses of FileUtil and functional correctness based on current use cases
[ https://issues.apache.org/jira/browse/HADOOP-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved HADOOP-8250. Resolution: Won't Fix Multiple other jiras have superceded this. > Investigate uses of FileUtil and functional correctness based on current use > cases > -- > > Key: HADOOP-8250 > URL: https://issues.apache.org/jira/browse/HADOOP-8250 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 1.1.0 >Reporter: Bikas Saha >Assignee: Bikas Saha > > The current Windows patch replaces symlink with copy. This jira tracks > understanding the implications of this change and others like it on expected > functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8536) Problem with -Dproperty=value option on windows hadoop
[ https://issues.apache.org/jira/browse/HADOOP-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved HADOOP-8536. Resolution: Duplicate Dup of HADOOP-8739 > Problem with -Dproperty=value option on windows hadoop > -- > > Key: HADOOP-8536 > URL: https://issues.apache.org/jira/browse/HADOOP-8536 > Project: Hadoop Common > Issue Type: Bug >Reporter: Trupti Dhavle > > While running the java examples the -Dproperty=value option to the hadoop > command is not getting read correctly. > TERASORT COMMAND: > C:\hdp\branch-1-win\bin\hadoop jar > C:\hdp\branch-1-win\build\hadoop-examples-1.1.0-SNAPSHOT.jar terasort > -Dmapreduce.reduce.input.limit=-1 teraInputDir teraOutputDir > Error- > 12/06/27 10:28:26 INFO terasort.TeraSort: starting > org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: > hdfs://localhost:8020/user/Administrator/-1 > It tries to look into directory named -1 instead of teraInputDir > On setting "echo on" in the cmd scripts, I noticed that the "=" sign > disappears in the command passed to JVM- > terasort -Dmapreduce.reduce.input.limit -1 teraInputDir teraOutputDir > In order to make it read properly quotes around “-Dproperty=value” are > required to be used. > This JIRA is to track fixing this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8739) Cmd scripts for Windows have issues in argument parsing
Bikas Saha created HADOOP-8739: -- Summary: Cmd scripts for Windows have issues in argument parsing Key: HADOOP-8739 URL: https://issues.apache.org/jira/browse/HADOOP-8739 Project: Hadoop Common Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha The parsing of the arguments has a bug in the way its broken down and this break things such at handling globbing (* char). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8664) hadoop streaming job need the full path to commands even when they are in the path
Bikas Saha created HADOOP-8664: -- Summary: hadoop streaming job need the full path to commands even when they are in the path Key: HADOOP-8664 URL: https://issues.apache.org/jira/browse/HADOOP-8664 Project: Hadoop Common Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha run a hadoop streaming job as bin/hadoop jar path_to_streaming_jar -input path_on_hdfs -mapper cat -output path_on_hdfs -reducer cat will fail saying program cat not found. cat is in the path and works from cmd prompt. If i give the full path to cmd.exe the exception is not seen -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8657) TestCLI fails on Windows because it uses hardcoded file length of test files committed to the source code
Bikas Saha created HADOOP-8657: -- Summary: TestCLI fails on Windows because it uses hardcoded file length of test files committed to the source code Key: HADOOP-8657 URL: https://issues.apache.org/jira/browse/HADOOP-8657 Project: Hadoop Common Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha The actual length of the file would depend on the character encoding used and hence cannot be hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8646) Avoid logging "Cannot run program getconf" on Windows
Bikas Saha created HADOOP-8646: -- Summary: Avoid logging "Cannot run program getconf" on Windows Key: HADOOP-8646 URL: https://issues.apache.org/jira/browse/HADOOP-8646 Project: Hadoop Common Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha ProcfsbasesProcessTree logs error messages when it cannot run getconf to determine system attributes on linux. this causes a lot of log spew on windows. need to fix this code because linux is not longer the only OS supported for hadoop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8645) Stabilize branch-1-win
Bikas Saha created HADOOP-8645: -- Summary: Stabilize branch-1-win Key: HADOOP-8645 URL: https://issues.apache.org/jira/browse/HADOOP-8645 Project: Hadoop Common Issue Type: Task Reporter: Bikas Saha Most of the code changes to make Hadoop branch 1 work natively on Windows are done. This jira in intended to track the work needed to achieve 100% test pass for the dev tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8618) Windows build failing after 1.0.3 got merged into branch-1-win
Bikas Saha created HADOOP-8618: -- Summary: Windows build failing after 1.0.3 got merged into branch-1-win Key: HADOOP-8618 URL: https://issues.apache.org/jira/browse/HADOOP-8618 Project: Hadoop Common Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8562) Enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments
Bikas Saha created HADOOP-8562: -- Summary: Enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments Key: HADOOP-8562 URL: https://issues.apache.org/jira/browse/HADOOP-8562 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8486) Resource leak - Close the open resource handles (File handles) before throwing the exception from the SequenceFile constructor
[ https://issues.apache.org/jira/browse/HADOOP-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved HADOOP-8486. Resolution: Fixed > Resource leak - Close the open resource handles (File handles) before > throwing the exception from the SequenceFile constructor > -- > > Key: HADOOP-8486 > URL: https://issues.apache.org/jira/browse/HADOOP-8486 > Project: Hadoop Common > Issue Type: Bug > Components: fs, io >Affects Versions: 1.0.2, 1-win >Reporter: Kanna Karanam >Assignee: Kanna Karanam > Fix For: 1-win > > Attachments: HADOOP-8486-branch-1-win-(2).patch, > HADOOP-8486-branch-1-win-(3).patch, HADOOP-8486-branch-1-win-(4).patch, > HADOOP-8486-branch-1-win-(5).patch, HADOOP-8486-branch-1-win.patch > > > I noticed this problem while I am working on porting HIVE to work on windows. > Hive is attempting to create this class object to validate the file format > and end up with resource leak. Because of this leak, we can’t move, rename or > delete the files on windows when there is an open file handle whereas in UNIX > we can perform all these operation with no issues even with open file handles. > Please suggest me if you similar issues in any other places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8409) Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for windows
[ https://issues.apache.org/jira/browse/HADOOP-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved HADOOP-8409. Resolution: Fixed > Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for > windows > - > > Key: HADOOP-8409 > URL: https://issues.apache.org/jira/browse/HADOOP-8409 > Project: Hadoop Common > Issue Type: Bug > Components: fs, test, util >Affects Versions: 1.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: HADOOP-8409-branch-1-win.2.patch, > HADOOP-8409-branch-1-win.3.patch, HADOOP-8409-branch-1-win.4.patch, > HADOOP-8409-branch-1-win.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > There are multiple places in prod and test code where Windows paths are not > handled properly. From a high level this could be summarized with: > 1. Windows paths are not necessarily valid DFS paths (while Unix paths are) > 2. Windows paths are not necessarily valid URIs (while Unix paths are) > #1 causes a number of tests to fail because they implicitly assume that local > paths are valid DFS paths (by extracting the DFS test path from for example > "test.build.data" property) > #2 causes issues when URIs are directly created on path strings passed in by > the user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8486) Resource leak - Close the open resource handles (File handles) before throwing the exception from the SequenceFile constructor
[ https://issues.apache.org/jira/browse/HADOOP-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha reopened HADOOP-8486: This has caused build failure because Path.getUriPath is no longer available. [javac] c:\Users\bikas\Code\hdp\src\test\org\apache\hadoop\io\TestSequenceFile.java:59: cannot find symbol [javac] symbol : method getUriPath() [javac] location: class org.apache.hadoop.fs.Path [javac] File f = new File(nonSeqFile.getUriPath()); > Resource leak - Close the open resource handles (File handles) before > throwing the exception from the SequenceFile constructor > -- > > Key: HADOOP-8486 > URL: https://issues.apache.org/jira/browse/HADOOP-8486 > Project: Hadoop Common > Issue Type: Bug > Components: fs, io >Affects Versions: 1.0.2, 1-win >Reporter: Kanna Karanam >Assignee: Kanna Karanam > Fix For: 1-win > > Attachments: HADOOP-8486-branch-1-win-(2).patch, > HADOOP-8486-branch-1-win-(3).patch, HADOOP-8486-branch-1-win-(4).patch, > HADOOP-8486-branch-1-win.patch > > > I noticed this problem while I am working on porting HIVE to work on windows. > Hive is attempting to create this class object to validate the file format > and end up with resource leak. Because of this leak, we can’t move, rename or > delete the files on windows when there is an open file handle whereas in UNIX > we can perform all these operation with no issues even with open file handles. > Please suggest me if you similar issues in any other places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8492) Rename ProcessTree.isSetsidAvailable
Bikas Saha created HADOOP-8492: -- Summary: Rename ProcessTree.isSetsidAvailable Key: HADOOP-8492 URL: https://issues.apache.org/jira/browse/HADOOP-8492 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 1.1.0 The logical use of this member is to find out if processes can be grouped into a unit for process manipulation. eg. killing process groups etc. setsid is the Linux implementation and it leaks into the name. I suggest renaming it to isProcessGroupAvailable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8424) Web UI broken on Windows because classpath not setup correctly
Bikas Saha created HADOOP-8424: -- Summary: Web UI broken on Windows because classpath not setup correctly Key: HADOOP-8424 URL: https://issues.apache.org/jira/browse/HADOOP-8424 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 1.1.0 The classpath is setup to include the hadoop jars before the build webapps directory and that upsets jetty when it is trying to resolve the webapp classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8421) Verify and fix build of c++ targets in Hadoop on Windows
Bikas Saha created HADOOP-8421: -- Summary: Verify and fix build of c++ targets in Hadoop on Windows Key: HADOOP-8421 URL: https://issues.apache.org/jira/browse/HADOOP-8421 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.0 Reporter: Bikas Saha Fix For: 1.1.0 There are a bunch of c++ files that are not compiled by default for legacy reasons. They represent important functionality. We need to make sure they build on Windows. There is some dependency on autoconf/autoreconf. HADOOP-8368 ideas could be used in here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8420) saveVersions.sh not working on Windows
Bikas Saha created HADOOP-8420: -- Summary: saveVersions.sh not working on Windows Key: HADOOP-8420 URL: https://issues.apache.org/jira/browse/HADOOP-8420 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.0 Reporter: Bikas Saha Fix For: 1.1.0 This script is executed during build time to generate version number information for Hadoop core. This version number is consumed via API's by Hive etc to determine compatibility with Hadoop versions. Currently, because of dependency on awk, cut etc utilities, this script does not run successfully and version information is not available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8401) Investigate use of JobObject to spawn tasks on Windows
Bikas Saha created HADOOP-8401: -- Summary: Investigate use of JobObject to spawn tasks on Windows Key: HADOOP-8401 URL: https://issues.apache.org/jira/browse/HADOOP-8401 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.0 Reporter: Bikas Saha -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8374) Improve support for hard link manipulation on Windows
Bikas Saha created HADOOP-8374: -- Summary: Improve support for hard link manipulation on Windows Key: HADOOP-8374 URL: https://issues.apache.org/jira/browse/HADOOP-8374 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 1.0.3 Hard link support for Windows does not work properly. There is some refactoring needed in the code. Also, the code currently executes the fsutil command to manipulate hard links. fsutil requires admin privileges on recent versions of Windows. The main features needed are the ability to create hard links to a file and count the number of hard links to a file. So we could use mklink to create hard links and write a custom executable to count hard links. Or use a custom executable to do both using Windows API's. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
