bq. run the tests claiming to require unaligned memory access on a platform
where unaligned memory access is definitely not supported for
shorts/ints/longs.
That would help us understand interactions on s390x platform better.
On Mon, Apr 18, 2016 at 6:49 AM, Adam Roberts <arobe...@uk.ibm.com> wrote:
> Ted, yes with the forced true value all tests pass, we use the unaligned
> check in 15 other suites.
>
> Our java.nio.Bits.unaligned() function checks that the detected os.arch
> value matches a list of known implementations (not including s390x).
>
> We could add it to the known architectures in the catch block but this
> won't make a difference here as because we call unaligned() OK (no
> exception is thrown), we don't reach the architecture checking stage anyway.
>
> I see in org.apache.spark.memory.MemoryManager that unaligned support is
> required for off-heap memory in Tungsten (perhaps incorrectly if no code
> ever exercises it in Spark?). Instead of having a requirement should we
> instead log a warning once that this is likely to lead to slow performance?
> What's the rationale for supporting unaligned memory access: it's my
> understanding that it's typically very slow, are there any design docs or
> perhaps a JIRA where I can learn more?
>
> Will run a simple test case exercising unaligned memory access for Linux
> on Z (without using Spark) and can also run the tests claiming to require
> unaligned memory access on a platform where unaligned memory access is
> definitely not supported for shorts/ints/longs.
>
> if these tests continue to pass then I think the Spark tests don't
> exercise unaligned memory access, cheers
>
>
>
>
>
>
>
> From: Ted Yu <yuzhih...@gmail.com>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "dev@spark.apache.org" <dev@spark.apache.org>
> Date: 15/04/2016 17:35
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> I am curious if all Spark unit tests pass with the forced true value for
> unaligned.
> If that is the case, it seems we can add s390x to the known architectures.
>
> It would also give us some more background if you can describe
> how java.nio.Bits#unaligned() is implemented on s390x.
>
> Josh / Andrew / Davies / Ryan are more familiar with related code. It
> would be good to hear what they think.
>
> Thanks
>
> On Fri, Apr 15, 2016 at 8:47 AM, Adam Roberts <*arobe...@uk.ibm.com*
> <arobe...@uk.ibm.com>> wrote:
> Ted, yeah with the forced true value the tests in that suite all pass and
> I know they're being executed thanks to prints I've added
>
> Cheers,
>
>
>
>
> From: Ted Yu <*yuzhih...@gmail.com* <yuzhih...@gmail.com>>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "*dev@spark.apache.org* <dev@spark.apache.org>" <
> *dev@spark.apache.org* <dev@spark.apache.org>>
> Date: 15/04/2016 16:43
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
> the forced true value for unaligned ?
>
> If the test failed, please pastebin the failure(s).
>
> Thanks
>
> On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <*arobe...@uk.ibm.com*
> <arobe...@uk.ibm.com>> wrote:
> Ted, yep I'm working from the latest code which includes that unaligned
> check, for experimenting I've modified that code to ignore the unaligned
> check (just go ahead and say we support it anyway, even though our JDK
> returns false: the return value of java.nio.Bits.unaligned()).
>
> My Platform.java for testing contains:
>
> private static final boolean unaligned;
>
> static {
> boolean _unaligned;
> // use reflection to access unaligned field
> try {
> * System.out.println("Checking unaligned support");*
> Class<?> bitsClass =
> Class.forName("java.nio.Bits", false,
> ClassLoader.getSystemClassLoader());
> Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
> unalignedMethod.setAccessible(true);
> _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
> *System.out.println("Used reflection and _unaligned is: " +
> _unaligned);*
> * System.out.println("Setting to true anyway for experimenting");*
> * _unaligned = true;*
> } catch (Throwable t) {
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> * // We don't actually get here since we find the unaligned method
> OK and it returns false (I override with true anyway)*
> * // but add s390x incase we somehow fail anyway.*
> * System.out.println("Checking for s390x, os.arch is: " + arch);*
> * _unaligned =
> arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");*
> }
> unaligned = _unaligned;
> * System.out.println("returning: " + unaligned);*
> }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
>
>
>
>
>
> From: Ted Yu <*yuzhih...@gmail.com* <yuzhih...@gmail.com>>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "*dev@spark.apache.org* <dev@spark.apache.org>" <
> *dev@spark.apache.org* <dev@spark.apache.org>>
> Date: 15/04/2016 15:32
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
>
> I assume you tested 2.0 with SPARK-12181 .
>
> Related code from Platform.java if java.nio.Bits#unaligned() throws
> exception:
>
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
>
> Can you give us some detail on how the code runs for JDKs on zSystems ?
>
> Thanks
>
> On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <*arobe...@uk.ibm.com*
> <arobe...@uk.ibm.com>> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> *core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java*
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> *core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java*
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> *common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java*
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
