Hi Namik, I finished most of the rearrangements for the BLAS3-benchmark, only a little bit of cleanup left (i.e. setting parameters in expert mode, looking into why the y-axis is only a few pixels high before pushing the first results). I'll start looking into deployment now.
Could you please have a look at 'your' items? Start with the ones you consider having highest priority to be completed for the first release :-) Thanks and best regards, Karli On 11/18/2014 10:13 PM, Namik Karovic wrote: > Hi Karl, > > thanks for the feedback. > > > -advanced mode: reset (load default) benchmark settings button > nice to have, but not a blocker :-) > > > Shouldn't be too hard to implement. I'll add it for the first release. > > > this is not as easy as it may seem. Even if the memory consumed by > the benchmark is below the physical memory provided, the GPU memory > may be occupied otherwise. This is particularly an issue with mobile > GPUs attached to 4K displays, where the screen buffer occupies a > decent amount of GPU RAM. I think the best we can do is to warn a > user, e.g. > "The benchmark configuration you provided exceeds XXX (256?) MB of > GPU RAM. This may exhaust your available GPU RAM. Do you want to > continue? [Continue]/[Cancel]" > > > A simple warning message is indeed a good solution. Added to my primary > TODO list. > > > > -vector benchmark progress update is slacking; it should be > updated with > each completed step instead of going from 0% to 100% in one step > (the > benchmarking loop needs to altered a little to fix this) > > > Fine with me. Can you fix this? > Also, I noticed that a repeated reset of the x-axis leads to ugly > fonts for the labels, so the axis should only be set once when > starting the benchmark. See screenshot attached, including some > other things... > > > A little rework in the benchmarking loop will fix the progress update > bug. Won't be an issue for me. > As for the x-axis resetting, new tick points are added with each result > entry which is why it looks so "jerky" while plotting. I'll make it so > that all tick points are added before the plotting begins. This should > also solve the font issue, since there won't be multiple tick labels > stacking on top of each other. > > > > -benchmarking with a custom sparse matrix now works with some > matrices, > but not all; at least now it doesn't crash the program... > > > This is presumably due to the matrix market format. Matrices with > scalar type 'complex' or with pattern type 'pattern' are not > supported. Can you add an error message stating this? Something like > "The matrix market reader cannot read the provided file. Only > real-valued matrices in coordinate-format are supported." > > > Does the matrix reader provide different error codes for different > reading errors? I see that it prints "supports only real valued floating > point arithmetic" to the console, so I should cover all possible error > types if there's more than one. > > > > -"my results" screen needs significant improvements:more > benchmark data > should be shown, auto-refresh when a result gets saved, > enable/disable > result saving, delete selected/all results, add user notes, maybe a > dynamic filter to search through result notes... > > > Can we disable the 'my results' screen for the first release? I > think it requires too much work *now*, but can be picked up any time > later. > > > -result uploading: it's completely missing > > > Disable that for the 1.0.0 release. The infrastructure is not yet > ready - in particular, result uploads are only interesting once we > have autotuning features in the GUI. > > > -"result database" screen: also completely missing > > > Disable :-) > > > Roger that. > > > > -"system info" screen feels kinda weird. Is it just me or maybe > it could > use some improvements? > > > Well, at least it is informative. ;-) > It would be great if you can make it a bit more appealing, but > visual appearance is not a top priority here I'd say... > > > Agreed. Moved to low priority, but I'll still keep it in my head in case > I think of something. > > > > -matrixmarket: reconnect the "download & run" functionality now that > benchmarking with custom sparse matrices does not crash the program. > Also, show the achieved result inside the matrixmarket table. > The idea > is to make it work like this: you find a nice matrix, hit download & > run, the program downloads the matrix, runs a benchmark with your > matrix, and shows the result next the "download&run" button. It > would be > good if we could avoid switching to the "benchmark" screen during > execution, and make it so that users don't need to leave the > "matrixmarket" screen at all. > > > Hmm, I think it is interesting for the user to see which sparse > matrix type works best for the particular sparse matrix. Unless we > can provide the full results from the 'sparse' tab in the > MatrixMarket, I'd rather refrain from it for now. > > > By "full results" do you mean those 7 tests that comprise the "sparse > benchmark" (HYB, ELL, CSR and others)? Showing 7 results instead of 1 > isn't a big a problem. I'd just have to play around the matrix table > layout to make 'em all fit properly. > > > > -some minor "face-lifting" procedures: Qt isn't very friendly > when it > comes to customizing the looks of widgets. That's why some > buttons and > tabs might look a bit wrong (depending on your platform). This > isn't a > significant issue, but still needs to be looked at. > > > I fixed a couple of these issues by removing hard-wired margins and > such. The home screen still has some layout issues, see screenshot. > > > About the hard-wired margins: when a result (bar graph) is too large to > fit on the plot, it goes out of bounds. Since the result value label is > "appended" to the end of the result bar, it goes out of bounds together > with the bar and cannot be displayed. The hard-wired margins were there > to ensure the labels will be displayed even if the result bar gets too > big. This kind of label rendering behavior is pretty weird if you ask > me, but that's how it is. Your solution to hard-code the axis range from > 0.1 to 5000 should work for now, but the labels will disappear if the > result gets close to that 5k mark. But I guess that's not a big problem > at the moment since 5k is currently not reachable (or is it?). > > As for the home screen layout issues and widget display bugs: like I > said, Qt turns into a spoiled brat as soon as you start messing with the > looks of its widgets. I can't say if it will be possible to make it look > good on all platforms. Lots of testing will be needed to ensure it looks > as it should on every platform... And that empty space in the home > screen should be definitely used for something, I just don't know what :D > > > Some more thoughts (some are trivial to fix): > - Result tabs: Can these tabs be renamed to "Dense > Matrix-Matrix-Products", "Host-Device Copy", "Sparse Matrix-Vector > Product", and "Vector Operations", respectively? I tried to do that, > but it seems like the current strings have some further meaning for > the program control flow (fix this?)? > > > You mean the detailed test results tab names in the bottom? I don't > recall their strings being important to the program flow control. Maybe > I hard-coded something for testing purposes but forgot to fix it > afterwards... > > > - I think it makes more sense for the BLAS 3 benchmark to scan > through different matrix sizes, rather than using a fixed size. The > benchmark just stops when a kernel run starts to take more than, > say, a second. This ensures that the benchmark does not get stuck on > mobile hardware, which is far slower than high-end desktop hardware > and for which we may not have good kernels yet. I'll take care of this. > > > Hmm, keep in mind that we need still need to make a "standardized" > benchmark configuration for the standard benchmark mode. Using different > matrix sizes for different hardware is not a very "standardized" thing > to do. > > - Results summary: Rather than providing an average, I think that > taking the best results makes more sense. For example, for the > sparse bench a user is really not interested in how many FLOPs are > obtained on average, but how many FLOPs the fastest matrix type > provides. Also an easy fix :-) > > > Heh, originally I made it so that the best result is shown in the > "result summary" graph. But then you said it might be better to use > median values so I went with that :) Twas probably a misunderstanding, > very easy to fix =) > > The final 'big TODO' is, of course, to provide a reasonably > automated packager to ship executables for Windows and Linux. I'm > less worried about Linux, because OpenCL can typically be found in a > system path. However, getting this to work with Windows might be > quite tricky, since a user might have the OpenCL lib installed at > whatever path. Either way, I reserved the weekend for this, so we > should be able to release early next week. (I can't postpone the > release, I'm totally filled with other work until the end of the > year then.) > > > Noob question: can't we package those OpenCL libs with our program, or > are those libs specific to each user's CPU/GPU manufacturer? What I'm > trying to ask is this: do users need to have a specific OpenCL SDK > installed (for example, I had to install AMD APP SDK separately for > development purposes), or is there a "general purpose" OpenCL lib that > anyone can use with any hardware and that also happens to be easily > package-able? > > As for an automated packager, we could use Qt Installer Framework > <http://qt-project.org/doc/qtinstallerframework-1.4/index.html>. I've > never used before, but considering the good documentation I don't think > it'll be a problem. > > > Regards, > Namik > > On Tue, Nov 18, 2014 at 10:46 AM, Karl Rupp <r...@iue.tuwien.ac.at > <mailto:r...@iue.tuwien.ac.at>> wrote: > > Hi Namik, > > thanks for the list. I'll comment on the items inline, particularly > with having the aim of a release in the near future in mind. > > > -advanced mode: reset (load default) benchmark settings button > > nice to have, but not a blocker :-) > > > -advanced mode: it would be cool if we could show how much video > memory > is required to complete the benchmark. This could be very useful > when > playing with custom settings as we would know in advance if we > set the > matrix size too high, thus preventing the program from crashing. > > > this is not as easy as it may seem. Even if the memory consumed by > the benchmark is below the physical memory provided, the GPU memory > may be occupied otherwise. This is particularly an issue with mobile > GPUs attached to 4K displays, where the screen buffer occupies a > decent amount of GPU RAM. I think the best we can do is to warn a > user, e.g. > "The benchmark configuration you provided exceeds XXX (256?) MB of > GPU RAM. This may exhaust your available GPU RAM. Do you want to > continue? [Continue]/[Cancel]" > > > -which makes me think a certain "safety switch" could also be a nice > addition, one that would prevent users from starting a benchmark > if the > settings are set too high. > > > I think the warning dialog above would take care of this. :-) > > > -vector benchmark progress update is slacking; it should be > updated with > each completed step instead of going from 0% to 100% in one step > (the > benchmarking loop needs to altered a little to fix this) > > > Fine with me. Can you fix this? > Also, I noticed that a repeated reset of the x-axis leads to ugly > fonts for the labels, so the axis should only be set once when > starting the benchmark. See screenshot attached, including some > other things... > > > -benchmarking with a custom sparse matrix now works with some > matrices, > but not all; at least now it doesn't crash the program... > > > This is presumably due to the matrix market format. Matrices with > scalar type 'complex' or with pattern type 'pattern' are not > supported. Can you add an error message stating this? Something like > "The matrix market reader cannot read the provided file. Only > real-valued matrices in coordinate-format are supported." > > > -"my results" screen needs significant improvements:more > benchmark data > should be shown, auto-refresh when a result gets saved, > enable/disable > result saving, delete selected/all results, add user notes, maybe a > dynamic filter to search through result notes... > > > Can we disable the 'my results' screen for the first release? I > think it requires too much work *now*, but can be picked up any time > later. > > > -result uploading: it's completely missing > > > Disable that for the 1.0.0 release. The infrastructure is not yet > ready - in particular, result uploads are only interesting once we > have autotuning features in the GUI. > > > -"result database" screen: also completely missing > > > Disable :-) > > > -"system info" screen feels kinda weird. Is it just me or maybe > it could > use some improvements? > > > Well, at least it is informative. ;-) > It would be great if you can make it a bit more appealing, but > visual appearance is not a top priority here I'd say... > > > -matrixmarket: reconnect the "download & run" functionality now that > benchmarking with custom sparse matrices does not crash the program. > Also, show the achieved result inside the matrixmarket table. > The idea > is to make it work like this: you find a nice matrix, hit download & > run, the program downloads the matrix, runs a benchmark with your > matrix, and shows the result next the "download&run" button. It > would be > good if we could avoid switching to the "benchmark" screen during > execution, and make it so that users don't need to leave the > "matrixmarket" screen at all. > > > Hmm, I think it is interesting for the user to see which sparse > matrix type works best for the particular sparse matrix. Unless we > can provide the full results from the 'sparse' tab in the > MatrixMarket, I'd rather refrain from it for now. > > > -some minor "face-lifting" procedures: Qt isn't very friendly > when it > comes to customizing the looks of widgets. That's why some > buttons and > tabs might look a bit wrong (depending on your platform). This > isn't a > significant issue, but still needs to be looked at. > > > I fixed a couple of these issues by removing hard-wired margins and > such. The home screen still has some layout issues, see screenshot. > > > Well, that's about all I can think of at the moment. As always, > you're > welcome to comment on these points and/or propose new things to > implement. > > P.S. If you would like me to change the visual appearance of > certain UI > parts, please say so before I start "fixing" those weird buttons and > tabs. Even the smallest changes can be a pain when it comes to > customizing Qt widgets. > > > Some more thoughts (some are trivial to fix): > - Result tabs: Can these tabs be renamed to "Dense > Matrix-Matrix-Products", "Host-Device Copy", "Sparse Matrix-Vector > Product", and "Vector Operations", respectively? I tried to do that, > but it seems like the current strings have some further meaning for > the program control flow (fix this?)? > > - I think it makes more sense for the BLAS 3 benchmark to scan > through different matrix sizes, rather than using a fixed size. The > benchmark just stops when a kernel run starts to take more than, > say, a second. This ensures that the benchmark does not get stuck on > mobile hardware, which is far slower than high-end desktop hardware > and for which we may not have good kernels yet. I'll take care of this. > > - Results summary: Rather than providing an average, I think that > taking the best results makes more sense. For example, for the > sparse bench a user is really not interested in how many FLOPs are > obtained on average, but how many FLOPs the fastest matrix type > provides. Also an easy fix :-) > > The final 'big TODO' is, of course, to provide a reasonably > automated packager to ship executables for Windows and Linux. I'm > less worried about Linux, because OpenCL can typically be found in a > system path. However, getting this to work with Windows might be > quite tricky, since a user might have the OpenCL lib installed at > whatever path. Either way, I reserved the weekend for this, so we > should be able to release early next week. (I can't postpone the > release, I'm totally filled with other work until the end of the > year then.) > > Best regards, > Karli > > ------------------------------------------------------------------------------ Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration & more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk _______________________________________________ ViennaCL-devel mailing list ViennaCL-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/viennacl-devel
