I am curious if all Spark unit tests pass with the forced true value for
unaligned.
If that is the case, it seems we can add s390x to the known architectures.
It would also give us some more background if you can describe how
java.nio.Bits#unaligned()
is implemented on s390x.
Josh / Andrew / Davies / Ryan are more familiar with related code. It would
be good to hear what they think.
Thanks
On Fri, Apr 15, 2016 at 8:47 AM, Adam Roberts <arobe...@uk.ibm.com> wrote:
> Ted, yeah with the forced true value the tests in that suite all pass and
> I know they're being executed thanks to prints I've added
>
> Cheers,
>
>
>
>
> From: Ted Yu <yuzhih...@gmail.com>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "dev@spark.apache.org" <dev@spark.apache.org>
> Date: 15/04/2016 16:43
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
> the forced true value for unaligned ?
>
> If the test failed, please pastebin the failure(s).
>
> Thanks
>
> On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <*arobe...@uk.ibm.com*
> <arobe...@uk.ibm.com>> wrote:
> Ted, yep I'm working from the latest code which includes that unaligned
> check, for experimenting I've modified that code to ignore the unaligned
> check (just go ahead and say we support it anyway, even though our JDK
> returns false: the return value of java.nio.Bits.unaligned()).
>
> My Platform.java for testing contains:
>
> private static final boolean unaligned;
>
> static {
> boolean _unaligned;
> // use reflection to access unaligned field
> try {
> * System.out.println("Checking unaligned support");*
> Class<?> bitsClass =
> Class.forName("java.nio.Bits", false,
> ClassLoader.getSystemClassLoader());
> Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
> unalignedMethod.setAccessible(true);
> _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
> *System.out.println("Used reflection and _unaligned is: " +
> _unaligned);*
> * System.out.println("Setting to true anyway for experimenting");*
> * _unaligned = true;*
> } catch (Throwable t) {
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> * // We don't actually get here since we find the unaligned method
> OK and it returns false (I override with true anyway)*
> * // but add s390x incase we somehow fail anyway.*
> * System.out.println("Checking for s390x, os.arch is: " + arch);*
> * _unaligned =
> arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");*
> }
> unaligned = _unaligned;
> * System.out.println("returning: " + unaligned);*
> }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
>
>
>
>
>
> From: Ted Yu <*yuzhih...@gmail.com* <yuzhih...@gmail.com>>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "*dev@spark.apache.org* <dev@spark.apache.org>" <
> *dev@spark.apache.org* <dev@spark.apache.org>>
> Date: 15/04/2016 15:32
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
>
> I assume you tested 2.0 with SPARK-12181 .
>
> Related code from Platform.java if java.nio.Bits#unaligned() throws
> exception:
>
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
>
> Can you give us some detail on how the code runs for JDKs on zSystems ?
>
> Thanks
>
> On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <*arobe...@uk.ibm.com*
> <arobe...@uk.ibm.com>> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> *core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java*
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> *core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java*
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> *common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java*
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
