Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
the forced true value for unaligned ?
If the test failed, please pastebin the failure(s).
Thanks
On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <arobe...@uk.ibm.com> wrote:
> Ted, yep I'm working from the latest code which includes that unaligned
> check, for experimenting I've modified that code to ignore the unaligned
> check (just go ahead and say we support it anyway, even though our JDK
> returns false: the return value of java.nio.Bits.unaligned()).
>
> My Platform.java for testing contains:
>
> private static final boolean unaligned;
>
> static {
> boolean _unaligned;
> // use reflection to access unaligned field
> try {
> * System.out.println("Checking unaligned support");*
> Class<?> bitsClass =
> Class.forName("java.nio.Bits", false,
> ClassLoader.getSystemClassLoader());
> Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
> unalignedMethod.setAccessible(true);
> _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
> *System.out.println("Used reflection and _unaligned is: " +
> _unaligned);*
> * System.out.println("Setting to true anyway for experimenting");*
> * _unaligned = true;*
> } catch (Throwable t) {
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> * // We don't actually get here since we find the unaligned method OK
> and it returns false (I override with true anyway)*
> * // but add s390x incase we somehow fail anyway.*
> * System.out.println("Checking for s390x, os.arch is: " + arch);*
> * _unaligned =
> arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");*
> }
> unaligned = _unaligned;
> * System.out.println("returning: " + unaligned);*
> }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
>
>
>
>
>
> From: Ted Yu <yuzhih...@gmail.com>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "dev@spark.apache.org" <dev@spark.apache.org>
> Date: 15/04/2016 15:32
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> I assume you tested 2.0 with SPARK-12181 .
>
> Related code from Platform.java if java.nio.Bits#unaligned() throws
> exception:
>
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
>
> Can you give us some detail on how the code runs for JDKs on zSystems ?
>
> Thanks
>
> On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <*arobe...@uk.ibm.com*
> <arobe...@uk.ibm.com>> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> *core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java*
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> *core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java*
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> *common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java*
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
