Re: [petsc-dev] https://www.dursi.ca/post/hpc-is-dying-and-mpi-is-killing-it.html

[Smith, Barry F. via petsc-dev]

> On May 16, 2019, at 6:16 PM, Mills, Richard Tran <rtmi...@anl.gov> wrote:
> 
> OK, so this thread is two months old but I saw some things recently that 
> reminded me of it.
> 
> To answer Barry's first question: I think that "AI" was used more than "HPC" 
> during the presentation because the HPC community's ridiculous focus on 
> rankings in the TOP500 list has resulted in machines that aren't truly good 
> for much other than xGEMM operations. And if you are looking around for 
> something to justify your xGEMM machine, well, deep neural nets fit the bill 
> pretty well.

   Is it actually a subset of deep neural networks? For example I can see deep 
neural networks that use convolutions as being good on GPUs because the amount 
of data that defines the network is very small and can stay near the compute 
unit (and has a good reuse since it is applied at each stencil point). On the 
other hand deep neural networks based on dense matrices are only BLAS level 2 
(dense matrix vector products) and are probably totally impractical anyways so 
won't be super good on GPUs, right? Meanwhile anything based on sparse matrices 
would give low performance. 

   Am I missing something?

> (Yes, the fact that GPUs are really good for, well, *graphics* and this is a 
> huge market -- way, way bigger than HPC -- is a contributing factor.)
> 
> On the health of HPC sales, I, like Bill, was thinking of what I see in 
> earnings reports from companies like Intel and NVIDIA. Yes, a lot of this is 
> driven by AI applications in data centers, but the same hardware gets used 
> for what I think of as more "traditional" HPC.
> 
> As for the increasing use of MPI in machine learning, two major examples are
> 
> * Uber's Horovod framework: https://eng.uber.com/horovod/
> * Microsoft's Cognitive Toolkit (CNTK) uses MPI for parallel training: 
> https://docs.microsoft.com/en-us/cognitive-toolkit/multiple-gpus-and-machines
> 
> There are other examples, too, but Uber and Microsoft are pretty big players. 
> I'm seeing a lot of examples of people using Horovod, in particular.
> 
> --Richard
> 
> On 3/19/19 4:11 PM, Gropp, William D wrote:
>> There is a sort of citation for the increasing use of MPI in distributed 
>> ML/DL - Torsten has a recent paper on demystifying ML with a graph based on 
>> published papers. Not the same, but interesting.
>> 
>> On the health of HPC, sales figures are available (along with attendance at 
>> SC) and these show HPC is healthy if not growing at unsustainable rates :)
>> 
>> Bill
>> 
>> 
>> On Mar 19, 2019 3:45 PM, "Smith, Barry F. via petsc-dev" 
>> <petsc-dev@mcs.anl.gov> wrote:
>> 
>> 
>> > On Mar 19, 2019, at 12:27 AM, Mills, Richard Tran via petsc-dev 
>> > <petsc-dev@mcs.anl.gov> wrote:
>> > 
>> > I've seen this quite some time ago. Others in this thread have already 
>> > articulated many of the same criticisms I have with the material in this 
>> > blog post, as well as some of the problems that I have with MPI, so I'll 
>> > content myself by asking the following: 
>> > 
>> > If HPC is as dying as this guy says it is, then
>> > 
>> > * Why did DOE just announce today that they are spending $500 million on 
>> > the first (there are *more* coming?) US-based exascale computing system?
>> 
>>    Why was the acronym AI used more often than HPC during the presentation?
>> 
>> > 
>> > * Why are companies like Intel, NVIDIA, Mellanox, etc., managing to sell 
>> > so much HPC hardware?
>> 
>>    Citation
>> > 
>> > and if it is all the fault of MPI, then
>> > 
>> > * Why have a bunch of the big machine-learning shops actually been moving 
>> > towards more use of MPI?
>> 
>>    Citation
>> 
>> > 
>> > Yeah, MPI has plenty of warts. So does Fortran -- yet that hasn't killed 
>> > scientific computing.
>> > 
>> > --Richard
>> > 
>> > On 3/17/19 1:12 PM, Smith, Barry F. via petsc-dev wrote:
>> >>   I stubbled on this today; I should have seen it years ago.
>> >> 
>> >>   Barry
>> >> 
>> >> 
>> > 
>> 
>> 
>