[
https://issues.apache.org/jira/browse/ARROW-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Wujciak-Jens updated ARROW-15368:
---------------------------------------
Description:
We should document the various env vars ({{{}ARROW_SIMD_LEVEL{}}},
{{{}ARROW_RUNTIME_SIMD_LEVEL{}}}, {{{}ARROW_USER_SIMD_LEVEL{}}}, others?).
We should also document what the defaults are (and what that means for
performance and possible optimization if you're compiling and you know you'll
be on more/less modern hardware:
e.g. pyarrow and the R package are compiled with SSE4_2, but there is some
amount of runtime dispatched simd code, and MAX there means that it will
compile everything it can. but at runtime it will use whatever is available. so
if you compile on a machine with AVX512 and run on a machine with AVX512,
you'll get any AVX512 runtime dispatched code that's available (probably not
much). There is more (esp. in the query engine) that is runtime AVX2.
FWIW I (neal) would leave ARROW_RUNTIME_SIMD_LEVEL=MAX always. You can set
ARROW_USER_SIMD_LEVEL to change/limit what level the runtime dispatch uses
Additionally we should document that valgrind does not support AVX512:
[https://bugs.kde.org/show_bug.cgi?id=383010]
And users should set ARROW_USER_SIMD_LEVEL to AVX2 if they plan to run valgrind
on an AVX512 capeable similar to what we do for our
[CI|https://github.com/apache/arrow/blob/bc1a16cd0eceeffe67893a7e8000d2dd28dcf3f1/docker-compose.yml#L309]
was:
We should document the various env vars ({{{}ARROW_SIMD_LEVEL{}}},
{{{}ARROW_RUNTIME_SIMD_LEVEL{}}}, {{{}ARROW_USER_SIMD_LEVEL{}}}, others?).
We should also document what the defaults are (and what that means for
performance and possible optimization if you're compiling and you know you'll
be on more/less modern hardware:
e.g. pyarrow and the R package are compiled with SSE4_2, but there is some
amount of runtime dispatched simd code, and MAX there means that it will
compile everything it can. but at runtime it will use whatever is available. so
if you compile on a machine with AVX512 and run on a machine with AVX512,
you'll get any AVX512 runtime dispatched code that's available (probably not
much). There is more (esp. in the query engine) that is runtime AVX2.
FWIW I (neal) would leave ARROW_RUNTIME_SIMD_LEVEL=MAX always. You can set
ARROW_USER_SIMD_LEVEL to change/limit what level the runtime dispatch uses
Additionally we should document that valgrind does not support AVX512:
[https://bugs.kde.org/show_bug.cgi?id=383010]
And users should set ARROW_USER_SIMD_LEVEL to AVX2 if they plan to run valgrind
similar to what we do for our
[CI|https://github.com/apache/arrow/blob/bc1a16cd0eceeffe67893a7e8000d2dd28dcf3f1/docker-compose.yml#L309]
> [C++] [Docs] Improve our SIMD documentation
> -------------------------------------------
>
> Key: ARROW-15368
> URL: https://issues.apache.org/jira/browse/ARROW-15368
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Documentation
> Reporter: Jonathan Keane
> Priority: Major
>
> We should document the various env vars ({{{}ARROW_SIMD_LEVEL{}}},
> {{{}ARROW_RUNTIME_SIMD_LEVEL{}}}, {{{}ARROW_USER_SIMD_LEVEL{}}}, others?).
> We should also document what the defaults are (and what that means for
> performance and possible optimization if you're compiling and you know you'll
> be on more/less modern hardware:
> e.g. pyarrow and the R package are compiled with SSE4_2, but there is some
> amount of runtime dispatched simd code, and MAX there means that it will
> compile everything it can. but at runtime it will use whatever is available.
> so if you compile on a machine with AVX512 and run on a machine with AVX512,
> you'll get any AVX512 runtime dispatched code that's available (probably not
> much). There is more (esp. in the query engine) that is runtime AVX2.
> FWIW I (neal) would leave ARROW_RUNTIME_SIMD_LEVEL=MAX always. You can set
> ARROW_USER_SIMD_LEVEL to change/limit what level the runtime dispatch uses
> Additionally we should document that valgrind does not support AVX512:
> [https://bugs.kde.org/show_bug.cgi?id=383010]
> And users should set ARROW_USER_SIMD_LEVEL to AVX2 if they plan to run
> valgrind on an AVX512 capeable similar to what we do for our
> [CI|https://github.com/apache/arrow/blob/bc1a16cd0eceeffe67893a7e8000d2dd28dcf3f1/docker-compose.yml#L309]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
