On 2 March 2013 03:33, Konstantin Boudnik <c...@apache.org> wrote: > > Windows is so different from _any_ Unix or pseudo-Unix flavors, including > Windows with Cygwin - that even multi-platform Java has hard hard time > dealing with it. This is enough, IMO, to warrant a separate checkpoint. > > Cygwin is the worst of both worlds: not Unix, not windows. Dropping it for proper windows is much better. Even dropping it altogether would be better. We hate cygwin problems in Ant as users have unrealistic expectations about the filesystem and how programs run.
CI-wise, it'd be good to have nightly builds and a preflight check per JIRA. It sounds like the consensus that is evolving is (in RFC 2119 terms) 1. CI test runs on Windows SHALL be provided (Matt has promised this) 2. A patch with Pass(Linux) && Fail(windows) tests MAY be committed 3. A patch with Pass(Linux) && Fail(windows) tests SHALL be fixed -but not necessarily by the author of the original patch 4. A patch with Pass(windows) && Fail(linux) tests MUST NOT be committed 5. * It is assumed that if it works on Linux, it SHOULD work on other Unix 6. A patch with Pass(other-unix) && Fail(linux) tests MUST NOT be committed (this has never arisen that I know of). This could be merged with (3) to state that: all patches must Pass(Linux). 7. The unix-side operation MAY BE optimised for Linux at the expense of other Unices (I remember that for exec()/fork() a way, way back). 8. The unix-side operation MAY contain features that only work on Linux (YARN-3 cgroups are an example of this) 9. A patch that is optimised for Linux SHOULD have a Windows alternative (c.f. local sockets). The alternative MAY be java code that substitutes for native C/C++/assembler That mention of native code raises another question: CPU support. Now that there's some ARM boxes running Hadoop on Jenkins, perhaps as AMD ramp up their ARM story and IBM continue with Power, and some new x86 ASM code is coming from Intel, we ought to have a policy there, something like 1. Hadoop MAY contain code that works best on x86 systems. 2. Hadoop MAY contain code that works best on ARM systems. 3. Hadoop MUST NOT contain code that only works on on a single CPU family 4. Hadoop SHOULD NOT contain code that works best on on a single CPU family AND makes performance worse on other CPU families. E.G we shouldn't mandate CRC, compression and encryption algorithms that speed up on one CPU family yet are significantly worse on other CPU families than a platform-neutral algorithm. 5. Hadoop MUST NOT contain code that has assumptions about the CPU memory model other than the java 5+ memory model [ http://www.cs.umd.edu/~pugh/java/memoryModel/] . This is automatic for Java code, but needs to be included in native code, as volatile is not a barrier operation in C/C++, some CPUs implement different barrier op 6. Hadoop MUST NOT contain code that has hard-code assumptions about cache other than CPUs implement coherency across cores. No hard coded assumptions about cache line sizes, write-through vs. write back. RAM could be NUMA, but MUST be consistent w.r.t. the causal model of the happens-before semantics of the java 5+ memory model. (if you didn't understand that, read up on memory models) That could be teased out for a separate discussion and vote, along with -maybe- a policy w.r.t non-HFDS filesystems (which could be SHOULD not break other FSs, MAY reduce performance, patches MAY break (without tests, who knows?). People implementing alternative filesystems and asserting compatibility with Hadoop SHOULD run their own CI tests.
