Re: [Vote] Merge branch-trunk-win to trunk
On Sun, Mar 3, 2013 at 8:50 PM, Harsh J ha...@cloudera.com wrote: Have we agreed (and stated it somewhere proper) that a -1 obtained for a Windows CI build for a test-patch will not block the ongoing work (unless it is Windows specific) and patches may still be committed to trunk despite that? This thread is long and possibly hard to follow. Yes, I and several others have stated that for now it is okay to commit even if Windows precommit build posts -1. I'm +1 if someone can assert and add the above into the formal guidelines. I'd still prefer that Windows does its releases separately as that ensures more quality for its audience and better testing periods (and wouldn't block anything), but we can come to that iff we are unable to maintain the currently proposed model. Which do you think is the right place to add this? At this time we are voting for merging into trunk. I prefer having a single release that supports both Linux and windows. Based on working on Windows support I think this is doable and should not hold up releases for Linux.
[jira] [Created] (HADOOP-9354) native.sln missing license header
Chris Nauroth created HADOOP-9354: - Summary: native.sln missing license header Key: HADOOP-9354 URL: https://issues.apache.org/jira/browse/HADOOP-9354 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Trivial We need to add the license header to native.sln. winutils.sln already has it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
RE: Building branch-1-win, Starting Datanode on Windows
I don't think you need to manually set permissions on Hadoop directories. Did you build the native windows executable in your build? Do you see any exceptions mentioning winutils? Bikas -Original Message- From: Daniel Jones [mailto:daniel.jo...@opencredo.com] Sent: Monday, March 04, 2013 6:56 AM To: common-dev@hadoop.apache.org Subject: Building branch-1-win, Starting Datanode on Windows Hi all, I've cloned the branch-1-win branch of Hadoop, and successfully managed to build it on my Windows 8 machine. When trying to start a datanode instance, I get the following error: 13/03/04 14:42:47 WARN datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for C:/hadoop/data, expected: rwxr-xr-x, while actual: --rwx 13/03/04 14:42:47 ERROR datanode.DataNode: All directories in dfs.data.dir are invalid. The directory exists, and I've set Allow all to Everyone. Is this an issue that should exist in the Windows branch? Is there a way to massage Windows' file permissions into something the datanode will accept without further modifications to the code? Many thanks in advance. -- Daniel Jones - Consultant Open Credo Ltd - Excellence in Enterprise Application Development Mobile: +44 (0)79 8000 9153 Main: +44 (0)20 3393 8242 daniel.jo...@opencredo.com http://twitter.com/BinaryTweedNet www.opencredo.com Registered Office: 5-11 Lavington Street, London, SE1 0NZ Registered in UK. No 3943999 If you have received this e-mail in error please accept our apologies, destroy it immediately and it would be greatly appreciated if you notified the sender. It is your responsibility to protect your system from viruses and any other harmful code or device. We try to eliminate them from e-mails and attachments; but we accept no liability for any that remain. We may monitor or access any or all e-mails sent to us.
[jira] [Resolved] (HADOOP-9354) Windows native project files missing license headers
[ https://issues.apache.org/jira/browse/HADOOP-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9354. - Resolution: Fixed Hadoop Flags: Reviewed +1. I committed the patch to branch-trunk-win. Thank you Chris! Windows native project files missing license headers Key: HADOOP-9354 URL: https://issues.apache.org/jira/browse/HADOOP-9354 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Trivial Attachments: HADOOP-9354-branch-trunk-win.1.patch, HADOOP-9354-branch-trunk-win.2.patch We need to add the license header to native.sln, native.vcxproj, and native.vcxproj.filters. The equivalent files in winutils already have the license headers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9355) Abstract symlink tests to use either FileContext or FileSystem
Andrew Wang created HADOOP-9355: --- Summary: Abstract symlink tests to use either FileContext or FileSystem Key: HADOOP-9355 URL: https://issues.apache.org/jira/browse/HADOOP-9355 Project: Hadoop Common Issue Type: Sub-task Reporter: Andrew Wang We'd like to run the symlink tests using both FileContext and the upcoming FileSystem implementation. The first step here is abstracting the test logic to run on an abstract filesystem implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9356) remove remaining references to cygwin/cygpath from scripts
Chris Nauroth created HADOOP-9356: - Summary: remove remaining references to cygwin/cygpath from scripts Key: HADOOP-9356 URL: https://issues.apache.org/jira/browse/HADOOP-9356 Project: Hadoop Common Issue Type: Improvement Components: build, scripts Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth branch-trunk-win still contains a few references to Cygwin and the cygpath command that need to be removed now that they are no longer needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.comwrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration (pre-commit build and unit test triggered by Jira Patch Available state) satisfy your request for functionality #1 and #2? Yes or no, please. Thanks, --Matt On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantin, I would like to explore what it would take to remove this perceived impediment -- Glad you decided to explore. Thank you. although I reserve the right to argue that this is not pre-requisite to merging the cross-platform support patch. It's your right indeed. So as mine to question what the platform support means for you, which I believe remained unclear. I do not impede the change as you should have noticed. My requirement comes from my perception of the support, which means to me exactly two things: 1. The ability to recognise the code is broken for the platform 2. The ability to test new patches on the platform The latter is problematic, as many noticed in this thread, for those whose customary environment does not include Windows. If we implemented full test-patch support for Windows on trunk, would that fulfill both your items #1 and #2? Please note that: a) Pushing the Patch Available button in Jira shall cause a pre-commit build to start within, I believe, 20 minutes. b) That build keeps logs for both java build and unit tests for several days, that are accessible to all viewers. In item #1 I mostly asking for the nightly build, which is simpler than test-patch. The latter would be ideal from the platform support viewpoint, but it is for the community to decide if we want to add extra +3 hours to the build. Nightly build in my understanding is triggered by the timer rather than by Jira's submit patch button. On Jenkins build configuration you can specify it under Build periodically. So, does this provide sufficient on-demand support that we don't have to implement a whole new on-demand VM support structure of some sort for #2 (which would be an extraordinary and impractical demand)? I did not mention VMs. Item #2 means a build, which runs test-patch target with the file specified by a user (instead of a jira attachment). When user clicks Build Now link a box is displayed where the user can enter the file path containing the patch. This can be specified in the Build Configuration under This build is parameterized by choosing AddParameter / FileParameter. The build can run on the same Windows machine as the nightly build. Such build will let people test their patches for Windows on Jenkins if they don't posses a license for the right version of Windows. I hope this will not turn into extraordinary or impractical effort. Thanks, --Konst Thanks, --Matt On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko shv.had...@gmail.com wrote: -1 We should have a CI infrastructure in place before we can commit to supporting Windows platform. Eric is right Win/Cygwin was supported since
RE: Building branch-1-win, Starting Datanode on Windows
Hi Daniel, This can happen when data directory permissions are different from default (755). Possible reasons could be: 1. This directory already exists but created with different permissions . One scenario I can think of : The directory got created as part of build workflow, and the permissions are inherited from parent (C:\Hadoop) 2. The datanode service tried to create this (if it does not exist) and failed while setting permissions using winutils. This case can be verified by looking at the DN log for FSShell exceptions To further investigate the root cause, it would help to find out who exactly created this directory. As Bikas mentioned, can you go through the logs for any exceptions related to permissions? Thanks, Ramya -Original Message- From: Bikas Saha [mailto:bi...@hortonworks.com] Sent: Monday, March 04, 2013 10:44 AM To: common-dev@hadoop.apache.org Subject: RE: Building branch-1-win, Starting Datanode on Windows I don't think you need to manually set permissions on Hadoop directories. Did you build the native windows executable in your build? Do you see any exceptions mentioning winutils? Bikas -Original Message- From: Daniel Jones [mailto:daniel.jo...@opencredo.com] Sent: Monday, March 04, 2013 6:56 AM To: common-dev@hadoop.apache.org Subject: Building branch-1-win, Starting Datanode on Windows Hi all, I've cloned the branch-1-win branch of Hadoop, and successfully managed to build it on my Windows 8 machine. When trying to start a datanode instance, I get the following error: 13/03/04 14:42:47 WARN datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for C:/hadoop/data, expected: rwxr-xr-x, while actual: --rwx 13/03/04 14:42:47 ERROR datanode.DataNode: All directories in dfs.data.dir are invalid. The directory exists, and I've set Allow all to Everyone. Is this an issue that should exist in the Windows branch? Is there a way to massage Windows' file permissions into something the datanode will accept without further modifications to the code? Many thanks in advance. -- Daniel Jones - Consultant Open Credo Ltd - Excellence in Enterprise Application Development Mobile: +44 (0)79 8000 9153 Main: +44 (0)20 3393 8242 daniel.jo...@opencredo.com http://twitter.com/BinaryTweedNet www.opencredo.com Registered Office: 5-11 Lavington Street, London, SE1 0NZ Registered in UK. No 3943999 If you have received this e-mail in error please accept our apologies, destroy it immediately and it would be greatly appreciated if you notified the sender. It is your responsibility to protect your system from viruses and any other harmful code or device. We try to eliminate them from e-mails and attachments; but we accept no liability for any that remain. We may monitor or access any or all e-mails sent to us.
Re: [Vote] Merge branch-trunk-win to trunk
On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration (pre-commit build and unit test triggered by Jira Patch Available state) satisfy your request for functionality #1 and #2? Yes or no, please. Thanks, --Matt On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantin, I would like to explore what it would take to remove this perceived impediment -- Glad you decided to explore. Thank you. although I reserve the right to argue that this is not pre-requisite to merging the cross-platform support patch. It's your right indeed. So as mine to question what the platform support means for you, which I believe remained unclear. I do not impede the change as you should have noticed. My requirement comes from my perception of the support, which means to me exactly two things: 1. The ability to recognise the code is broken for the platform 2. The ability to test new patches on the platform The latter is problematic, as many noticed in this thread, for those whose customary environment does not include Windows. If we implemented full test-patch support for Windows on trunk, would that fulfill both your items #1 and #2? Please note that: a) Pushing the Patch Available button in Jira shall cause a pre-commit build to start within, I believe, 20 minutes. b) That build keeps logs for both java build and unit tests for several days, that are accessible to all viewers. In item #1 I mostly asking for the nightly build, which is simpler than test-patch. The latter would be ideal from the platform support viewpoint, but it is for the community to decide if we want to add extra +3 hours to the build. Nightly build in my understanding is triggered by the timer rather than by Jira's submit patch button. On Jenkins build configuration you can specify it under Build periodically. So, does this provide sufficient on-demand support that we don't have to implement a whole new on-demand VM support structure of some sort for #2 (which would be an extraordinary and impractical demand)? I did not mention VMs. Item #2 means a build, which runs test-patch target with the file specified by a user (instead of a jira attachment). When user clicks Build Now link a box is displayed where the user can enter the file
[jira] [Created] (HADOOP-9358) Auth failed log should include exception string
Todd Lipcon created HADOOP-9358:
---
Summary: Auth failed log should include exception string
Key: HADOOP-9358
URL: https://issues.apache.org/jira/browse/HADOOP-9358
Project: Hadoop Common
Issue Type: Bug
Components: ipc, security
Affects Versions: 3.0.0, 2.0.4-beta
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Currently, when authentication fails, we see a WARN message like:
{code}
2013-02-28 22:49:03,152 WARN ipc.Server (Server.java:saslReadAndProcess(1056))
- Auth failed for 1.2.3.4:12345:null
{code}
This is not useful to understand the underlying cause. The WARN entry should
additionally include the exception text, eg:
{code}
2013-02-28 22:49:03,152 WARN ipc.Server (Server.java:saslReadAndProcess(1056))
- Auth failed for 1.2.3.4:12345:null (GSS initiate failed [Caused by
GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is
a replay (34))])
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Thanks. I agree Windows -1's in test-patch should not block commits. --Matt On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko shv.had...@gmail.comwrote: On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration (pre-commit build and unit test triggered by Jira Patch Available state) satisfy your request for functionality #1 and #2? Yes or no, please. Thanks, --Matt On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantin, I would like to explore what it would take to remove this perceived impediment -- Glad you decided to explore. Thank you. although I reserve the right to argue that this is not pre-requisite to merging the cross-platform support patch. It's your right indeed. So as mine to question what the platform support means for you, which I believe remained unclear. I do not impede the change as you should have noticed. My requirement comes from my perception of the support, which means to me exactly two things: 1. The ability to recognise the code is broken for the platform 2. The ability to test new patches on the platform The latter is problematic, as many noticed in this thread, for those whose customary environment does not include Windows. If we implemented full test-patch support for Windows on trunk, would that fulfill both your items #1 and #2? Please note that: a) Pushing the Patch Available button in Jira shall cause a pre-commit build to start within, I believe, 20 minutes. b) That build keeps logs for both java build and unit tests for several days, that are accessible to all viewers. In item #1 I mostly asking for the nightly build, which is simpler than test-patch. The latter would be ideal from the platform support viewpoint, but it is for the community to decide if we want to add extra +3 hours to the build. Nightly build in my understanding is triggered by the timer rather than by Jira's submit patch button. On Jenkins build configuration you can specify it under Build periodically. So, does this provide sufficient on-demand support that we don't have to implement a whole new on-demand VM support structure of some sort for #2
[jira] [Resolved] (HADOOP-9356) remove remaining references to cygwin/cygpath from scripts
[ https://issues.apache.org/jira/browse/HADOOP-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9356. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed I committed the patch to branch-trunk-win. Thank you Chris! remove remaining references to cygwin/cygpath from scripts -- Key: HADOOP-9356 URL: https://issues.apache.org/jira/browse/HADOOP-9356 Project: Hadoop Common Issue Type: Improvement Components: build, scripts Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: HADOOP-9356-branch-trunk-win.1.patch, HADOOP-9356-branch-trunk-win.2.patch branch-trunk-win still contains a few references to Cygwin and the cygpath command that need to be removed now that they are no longer needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Ok, looks like we are converging on this across a few hundred emails ;) So, as has been stated elsewhere: test-patch will be improved to fully support Windows; furthermore -1 from Windows' test-patch won't block Linux commits. This is ok with me. Can we have a JIRA ticket for that test-patch work assigned to the real owner, so it can be tracked? I am +1 in this case. Cos On Fri, Mar 01, 2013 at 04:57PM, Chris Douglas wrote: On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Commitment is a good thing. I think the two builds that I proposed are a prerequisite for Win support. If we commit windows patch people will start breaking it the next day. Which we wont know without the nightly build and wont be able to fix without the on-demand one. As several people have pointed out already, the surface of possible conflicts is relatively limited, and- as you did in 2007- the devs on Windows will report and fix bugs in that platform as they find them. CI is important for detecting and preventing bugs, but this isn't software we're launching into orbit. Making two builds is less than 2 days work, imho, given that there is a Windows node available and that mvn targets are in place. Correct me if I missed any complications in the process. On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik c...@apache.org wrote: It seems that with the HW in place, the matter of setting at least nightly build is trivial for anyone with up to date Windows knowledge. I wish I could help. Going without a validation is a recipe for a disaster IMO. Fair enough, though that also implies that the window for regressions is small, and it leaves little room to doubt that this will receive priority. Until it's merged, spurious notifications that the current trunk breaks Windows are an awkward introduction to devs' workflow. The order of merge/CI is a choice between mild annoyances, really. But it might be moot. Giri: you're doing the work on this. When do you think it can be complete? -C
Re: [Vote] Merge branch-trunk-win to trunk
+1 on the merge. I am glad we agreed. Having Jira to track the CI effort is a good idea. Thanks, --Konstantin On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley mfo...@hortonworks.com wrote: Thanks. I agree Windows -1's in test-patch should not block commits. --Matt On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko shv.had...@gmail.com wrote: On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration (pre-commit build and unit test triggered by Jira Patch Available state) satisfy your request for functionality #1 and #2? Yes or no, please. Thanks, --Matt On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantin, I would like to explore what it would take to remove this perceived impediment -- Glad you decided to explore. Thank you. although I reserve the right to argue that this is not pre-requisite to merging the cross-platform support patch. It's your right indeed. So as mine to question what the platform support means for you, which I believe remained unclear. I do not impede the change as you should have noticed. My requirement comes from my perception of the support, which means to me exactly two things: 1. The ability to recognise the code is broken for the platform 2. The ability to test new patches on the platform The latter is problematic, as many noticed in this thread, for those whose customary environment does not include Windows. If we implemented full test-patch support for Windows on trunk, would that fulfill both your items #1 and #2? Please note that: a) Pushing the Patch Available button in Jira shall cause a pre-commit build to start within, I believe, 20 minutes. b) That build keeps logs for both java build and unit tests for several days, that are accessible to all viewers. In item #1 I mostly asking for the nightly build, which is simpler than test-patch. The latter would be ideal from the platform support viewpoint, but it is for the community to decide if we want to add extra +3 hours to the build. Nightly build in my understanding is triggered by the timer rather than by Jira's submit patch button. On Jenkins build configuration
Re: Building branch-1-win, Starting Datanode on Windows
Daniel, Could you pls check if you have 64 bit vc runtime and dotnet framework installed? 64 bit vc runtime is critical. -Giri On Mon, Mar 4, 2013 at 12:40 PM, Ramya Nimmagadda ramya.nimmaga...@microsoft.com wrote: Hi Daniel, This can happen when data directory permissions are different from default (755). Possible reasons could be: 1. This directory already exists but created with different permissions . One scenario I can think of : The directory got created as part of build workflow, and the permissions are inherited from parent (C:\Hadoop) 2. The datanode service tried to create this (if it does not exist) and failed while setting permissions using winutils. This case can be verified by looking at the DN log for FSShell exceptions To further investigate the root cause, it would help to find out who exactly created this directory. As Bikas mentioned, can you go through the logs for any exceptions related to permissions? Thanks, Ramya -Original Message- From: Bikas Saha [mailto:bi...@hortonworks.com] Sent: Monday, March 04, 2013 10:44 AM To: common-dev@hadoop.apache.org Subject: RE: Building branch-1-win, Starting Datanode on Windows I don't think you need to manually set permissions on Hadoop directories. Did you build the native windows executable in your build? Do you see any exceptions mentioning winutils? Bikas -Original Message- From: Daniel Jones [mailto:daniel.jo...@opencredo.com] Sent: Monday, March 04, 2013 6:56 AM To: common-dev@hadoop.apache.org Subject: Building branch-1-win, Starting Datanode on Windows Hi all, I've cloned the branch-1-win branch of Hadoop, and successfully managed to build it on my Windows 8 machine. When trying to start a datanode instance, I get the following error: 13/03/04 14:42:47 WARN datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for C:/hadoop/data, expected: rwxr-xr-x, while actual: --rwx 13/03/04 14:42:47 ERROR datanode.DataNode: All directories in dfs.data.dir are invalid. The directory exists, and I've set Allow all to Everyone. Is this an issue that should exist in the Windows branch? Is there a way to massage Windows' file permissions into something the datanode will accept without further modifications to the code? Many thanks in advance. -- Daniel Jones - Consultant Open Credo Ltd - Excellence in Enterprise Application Development Mobile: +44 (0)79 8000 9153 Main: +44 (0)20 3393 8242 daniel.jo...@opencredo.com http://twitter.com/BinaryTweedNet www.opencredo.com Registered Office: 5-11 Lavington Street, London, SE1 0NZ Registered in UK. No 3943999 If you have received this e-mail in error please accept our apologies, destroy it immediately and it would be greatly appreciated if you notified the sender. It is your responsibility to protect your system from viruses and any other harmful code or device. We try to eliminate them from e-mails and attachments; but we accept no liability for any that remain. We may monitor or access any or all e-mails sent to us.
[jira] [Created] (HADOOP-9359) Add Windows build and unit test to test-patch pre-commit testing
Matt Foley created HADOOP-9359: -- Summary: Add Windows build and unit test to test-patch pre-commit testing Key: HADOOP-9359 URL: https://issues.apache.org/jira/browse/HADOOP-9359 Project: Hadoop Common Issue Type: Sub-task Reporter: Matt Foley Assignee: Matt Foley The test-patch utility is triggered by Patch Available state in Jira, and runs nine different sets of builds, tests, and static analysis tools. Currently only the Linux environment is tested. Need to add tests for Java build under Windows, and unit test execution under Windows. At this time, the community has decided that -1 on these new additional tests shall not block commits to the code base. However, contributors and code reviewers are encouraged to utilize the information provided by these tests to help keep Hadoop cross-platform compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Thanks, gentlemen. I've opened and taken responsibility for https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has agreed to help with the parts that require Jenkins admin access. Thanks, --Matt On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko shv.had...@gmail.comwrote: +1 on the merge. I am glad we agreed. Having Jira to track the CI effort is a good idea. Thanks, --Konstantin On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley mfo...@hortonworks.com wrote: Thanks. I agree Windows -1's in test-patch should not block commits. --Matt On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko shv.had...@gmail.com wrote: On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration (pre-commit build and unit test triggered by Jira Patch Available state) satisfy your request for functionality #1 and #2? Yes or no, please. Thanks, --Matt On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantin, I would like to explore what it would take to remove this perceived impediment -- Glad you decided to explore. Thank you. although I reserve the right to argue that this is not pre-requisite to merging the cross-platform support patch. It's your right indeed. So as mine to question what the platform support means for you, which I believe remained unclear. I do not impede the change as you should have noticed. My requirement comes from my perception of the support, which means to me exactly two things: 1. The ability to recognise the code is broken for the platform 2. The ability to test new patches on the platform The latter is problematic, as many noticed in this thread, for those whose customary environment does not include Windows. If we implemented full test-patch support for Windows on trunk, would that fulfill both your items #1 and #2? Please note that: a) Pushing the Patch Available button in Jira shall cause a pre-commit build to start within, I believe, 20 minutes. b) That build keeps logs for both java build and unit tests for
Re: [Vote] Merge branch-trunk-win to trunk
Thanks Suresh. Regarding where; we can state it on http://wiki.apache.org/hadoop/HowToContribute in the test-patch section perhaps. +1 on the merge. On Mon, Mar 4, 2013 at 11:39 PM, Suresh Srinivas sur...@hortonworks.com wrote: On Sun, Mar 3, 2013 at 8:50 PM, Harsh J ha...@cloudera.com wrote: Have we agreed (and stated it somewhere proper) that a -1 obtained for a Windows CI build for a test-patch will not block the ongoing work (unless it is Windows specific) and patches may still be committed to trunk despite that? This thread is long and possibly hard to follow. Yes, I and several others have stated that for now it is okay to commit even if Windows precommit build posts -1. I'm +1 if someone can assert and add the above into the formal guidelines. I'd still prefer that Windows does its releases separately as that ensures more quality for its audience and better testing periods (and wouldn't block anything), but we can come to that iff we are unable to maintain the currently proposed model. Which do you think is the right place to add this? At this time we are voting for merging into trunk. I prefer having a single release that supports both Linux and windows. Based on working on Windows support I think this is doable and should not hold up releases for Linux. -- Harsh J
Re: [Vote] Merge branch-trunk-win to trunk
Added to the Jira to modify http://wiki.apache.org/hadoop/HowToContribute to document this decision. On Mon, Mar 4, 2013 at 5:42 PM, Harsh J ha...@cloudera.com wrote: Thanks Suresh. Regarding where; we can state it on http://wiki.apache.org/hadoop/HowToContribute in the test-patch section perhaps. +1 on the merge. On Mon, Mar 4, 2013 at 11:39 PM, Suresh Srinivas sur...@hortonworks.com wrote: On Sun, Mar 3, 2013 at 8:50 PM, Harsh J ha...@cloudera.com wrote: Have we agreed (and stated it somewhere proper) that a -1 obtained for a Windows CI build for a test-patch will not block the ongoing work (unless it is Windows specific) and patches may still be committed to trunk despite that? This thread is long and possibly hard to follow. Yes, I and several others have stated that for now it is okay to commit even if Windows precommit build posts -1. I'm +1 if someone can assert and add the above into the formal guidelines. I'd still prefer that Windows does its releases separately as that ensures more quality for its audience and better testing periods (and wouldn't block anything), but we can come to that iff we are unable to maintain the currently proposed model. Which do you think is the right place to add this? At this time we are voting for merging into trunk. I prefer having a single release that supports both Linux and windows. Based on working on Windows support I think this is doable and should not hold up releases for Linux. -- Harsh J
[jira] [Resolved] (HADOOP-9232) JniBasedUnixGroupsMappingWithFallback fails on Windows with UnsatisfiedLinkError
[
https://issues.apache.org/jira/browse/HADOOP-9232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9232.
-
Resolution: Fixed
Fix Version/s: trunk-win
Hadoop Flags: Reviewed
Committed the patch to the branch.
Thank you Ivan!. Thank you Arpit, Chuan and Chris for the review.
JniBasedUnixGroupsMappingWithFallback fails on Windows with
UnsatisfiedLinkError
Key: HADOOP-9232
URL: https://issues.apache.org/jira/browse/HADOOP-9232
Project: Hadoop Common
Issue Type: Bug
Components: native, security
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Ivan Mitic
Fix For: trunk-win
Attachments: HADOOP-9232.branch-trunk-win.jnigroups.2.patch,
HADOOP-9232.branch-trunk-win.jnigroups.3.patch,
HADOOP-9232.branch-trunk-win.jnigroups.patch, HADOOP-9232.patch
{{JniBasedUnixGroupsMapping}} calls native code which isn't implemented
properly for Windows, causing {{UnsatisfiedLinkError}}. The fallback logic
in {{JniBasedUnixGroupsMappingWithFallback}} works by checking if the native
code is loaded during startup. In this case, hadoop.dll is present and
loaded, but it doesn't contain the right code. There will be no attempt to
fallback to {{ShellBasedUnixGroupsMapping}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9250) Windows installer bugfixes
[ https://issues.apache.org/jira/browse/HADOOP-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9250. - Resolution: Fixed Fix Version/s: 1-win Hadoop Flags: Reviewed +1. I committed the patch to branch-1-win. Thank you Ivan! Thanks to [~kanna...@microsoft.com] for the review. Windows installer bugfixes -- Key: HADOOP-9250 URL: https://issues.apache.org/jira/browse/HADOOP-9250 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 1-win Attachments: HADOOP-9250.branch-1-win.installerbugs.patch A few bugfixes and improvements we made to the install scripts on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
