The *annum mirabilis* "exceeds expectations", that's beautiful :-) Peter
On Tue, Nov 26, 2013 at 9:15 AM, Lux, Jim (337C) <[email protected]>wrote: > > On 11/25/13 4:11 PM, "Adam DeConinck" <[email protected]> wrote: > > >-----BEGIN PGP SIGNED MESSAGE----- > >Hash: SHA512 > > > >> 4. I went to a BoF on ROI on HPC investment. All the presentations in > >> the BoF frustrated me. Not because they were poorly done, but because > >> they tried to measure the value of a cluster by number of papers > >> published that used that HPC resource. I think that's a crappy, crappy > >> metric, but haven't been able to come up with a better one myself yet. > >>I > >> was very vocal with my comments and criticisms of the presentations, so > >> if any of the presenters are reading this now, I apologize for > >> hi-jacking your BoF. Getting good ROI on a cluster is close to my > >>heart, > >> but is also difficult to quantify and measure. I hope I can be part of > >> the discussion next year. > >> > > > >Do you have any thoughts you can share on what alternative metrics > >might look like, even if you can't think of one that's clearly better? > > > >I have no horse in this race as I've been doing industry HPC for the > >past few years, but I'm curious what good metrics for ROI on an academic > >or lab cluster might be. Total number of papers? Number of > >citations after an N-year time window? [shrug] > > > >ROI measurement can sometimes be difficult even in an industrial or > >commercial setting, especially if the HPC resource is used for R&D or > >"engineering support" as opposed to something that feeds directly into > >the product. > > > >Cheers, > >Adam > > Definitely a challenge. > > Maybe we have webcams that look at all the users and we calculate > percentage of time smiling while interacting with the cluster? > > ROI for "technology development" is a tough thing to calculate. ROI, by > it's nature is a "money returned for money spent", and the return is > somewhat intangible. > > All of these metrics require having a baseline so you can do a > before/after comparison. And realistically, there needs to be a fairly > long averaging time on the metric. Here's the annual paper output of a > noted physicist. > > 1901 1 > 1902 2 > 1903 1 > 1904 1 > 1905 25 > 1906 6 > 1907 8 > 1908 4 > 1909 5 > 1910 6 > 1911 8 > > How would you evaluate the ROI of feeding him? Started kind of slow, had > a really good year, and likely received a "exceeds expectations" annual > review. But that set a new bar, and now his supervisor is going to be > hammering him.. Dude, your output is slacking off, I think we need to put > you on a performance improvement plan, and this year, you're going to be > "does not meet" in your review. > > > The other problem is that paper publishing (and the schedule thereof) is > influenced by things other than availability of computational resources, > so you need a very large sample so those influences average out. For > instance, lack of funds or permission to travel to a conference, combined > with the recent fad of "you must present in person" will have an effect. > The sequester and/or furlough will almost certainly manifest itself in any > sort of time series counting publications. > > The other thing is that there is a long gestation period for some work. > You might not have something "publishable", especially with the bias > against publishing null or negative results. That doesn't mean that the > HPC work wasn't useful, if it found a bunch of "ways not to go". > > There might also be a availability of workforce to grind out the papers > issue. At least at JPL, relatively few people work on a single job or > task. A more typical scenario is having 2 or 3 projects you work on > simultaneously, along with half a dozen things you support. There is a > tendency to spend one's time on the latest thing to go wrong, and in a "do > more with less" environment, there's not a lot of down time in which to > catch up. > > In an environment where short term results are more important (or, at > least have more "gain" in the control loop) it's tough to push "getting > published" higher up the priority list, since the personal ill effects of > not publishing may be years down the road, compared to immediate ill > effects of "the project will be cancelled if we don't make the deliverable > this month". > > At JPL, it is easy to tell in which organizations, the "papers published" > metric is important in annual ranking and review: they're the ones with > lots of papers. That's not to say that other organizations don't do lots > of publishable work, but if your annual review depends on something > *other* than the metric, you're not going to spend your time doing it. > > Export controls also rear their ugly head. A lot of interesting problems > that can be attacked by HPC are the practical ones. But in a number of > industries, once you move beyond pure research and theory (TRL 3), you get > into an area where it is either competition sensitive or export > controlled. It is true that you could write a paper that is suitably > expurgated and sanitized, but you still have to go through the export > control/public release review process. And that's time consuming too. > > > The whole competition sensitive/proprietary rights/ export controls aspect > might be why some of you have commented on the gulf between what's > presented on the show floor and what's presented in the talks. > > _______________________________________________ > Beowulf mailing list, [email protected] sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf >
_______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
