I like those ideas. Filed https://issues.apache.org/jira/browse/IMPALA-6166
On Mon, Nov 6, 2017 at 11:26 AM, Todd Lipcon <t...@cloudera.com> wrote: > FWIW in Kudu we draw the line at requiring SSE4.2 but not AVX. I think this > means we support Westmere (2010) and potentially also 1st gen Nehalem > (2008). Going back older than 2008 definitely seems excessively generous. > > Another thing to keep in mind is that some virtualization software may not > pass through all instruction set extensions. It would be worth double > checking that relatively recent versions of VirtualBox or other > commonly-used desktop VM options pass through AVX properly before making it > a requirement. > > -Todd > > On Mon, Nov 6, 2017 at 10:33 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > >> I don't believe that we've seen people "seriously" using Impala with older >> hardware than that recently, but it's hard to know for sure. >> >> It's nice for adoption to support relatively old hardware - one thing we >> have seen before is people installing Impala on an old cluster to try it >> out. E.g. they wanted to try Impala and had four old servers sitting around >> unused. I don't think we should optimise too much for that case, but it's >> an argument for supporting ~5 year old hardware that has probably been >> retired from its original purpose. >> >> One option we could consider is drawing the line at >> https://en.wikipedia.org/wiki/Sandy_Bridge and >> https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)# >> Instruction_set_extensions. >> That would give us CLMUL, POPCNT, SSSE3, SSE4.1, SSE4.2 and AVX. It doesn't >> look like AVX2 was available on AMD chips until 2015. >> >> It seems less disruptive if we drop support for older processors in a major >> release - i.e. Impala 3.0. I don't think that needs to be a strict policy >> of never dropping hardware support in a minor release but I think it's more >> convenient for users. >> >> On Sat, Nov 4, 2017 at 10:53 PM, Jim Apple <jbap...@cloudera.com> wrote: >> >> > In a discussion on https://issues.apache.org/jira/browse/IMPALA-6128, >> > we are talking about which instruction sets (available on newer x86-64 >> > processors) we want to require. >> > >> > At this point, I'm not sure how strong the motivation is for requiring >> > certain instruction sets, but it may be worth some effort to talk >> > about guidelines. As of now, we can decide at run time which methods >> > to use based on CPU info gathered at daemon start time. See >> > cpu-info.cc. >> > >> > The instruction in this case is the CLMUL instruction, which we >> > believe was available on all new server-class x86-64 chips by Intel >> > and AMD as of Q2, 2011. It has good performance benefits for >> > spill-to-disk encryption. >> > >> > We currently use the following, but only dispatching at run time: >> > >> > SSSE3(*), SSE4.1, SSE4.2 (Available since late 2011 on both AMD and >> Intel) >> > POPCNT (Available since late 2008 on both AMD and Intel) >> > AVX (late 2011) >> > AVX2 (late 2015) >> > >> > One argument for continuing with our current requirements is that >> > dispatching still gets us good speedup in some cases, and the branch >> > predictor should take care of some of the latency of dispatching. >> > >> > One argument for adding more requirements is that not only can >> > dispatching go away, but we can add flags to the compilers to use >> > later instructions, which can speed up auto-vectorized operations or >> > standard library operations. For instance, AVX has 256-bit registers >> > that can speed up bulk memory operations. >> > >> > A concern I have with setting a time-based rule is that it doesn't >> > seem easy to me to figure out when, say, AMD *stopped* selling >> > server-class chips without AVX. So, if we started requiring AVX, we >> > could have some Impala user with recent AMD chips become unable to run >> > the latest Impala, which would be a shame. >> > >> > Thoughts about what we should require? >> > >> > (*) We spit out an error if the machine does not have SSSE3 >> > >> > > > > -- > Todd Lipcon > Software Engineer, Cloudera
