Yes, SSD will definitely help. Just keep in mind, it doesn't prevent all negatives in regards to I/O, particularly with regards to caching.
Disk caching in a modern system is fairly complex, but at the high level it is not only done by the controller, but by the OS as well. So randomly flying around the disk still cause cache thrashing. :( -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: Tuesday, October 02, 2012 11:19 AM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe Great point!! I think we can agree that 'spinning media latency' is the enemy and having phantoms increasing the 'head dance' can make things worse, not better! Many problems go away or become trivial as the spinning media trails to the sunset. I've advised customers that just moving 'code files' to a tiny SSD would likely increase overall system performance on Windows boxes. Just waiting until the price for Enterprise SSDs makes them a no-brainer... Until then, even small SSDs will help! -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Daniel McGrath Sent: Tuesday, October 02, 2012 12:05 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe You've highlighted one problem here. By having multiple processes accessing the disk in different locations, you destroy cache optimization and seek times. More phantoms = less performance. This assumes I/O is a bigger concern than CPU, which is generally the case. More phantoms = more communication, which also adds another overhead that reduces performance. Introducing more phantoms than CPU cores, you increase the amount of context switching, which ones again hurts your cache usage as well as adding bigger overheads on the CPU again. In short, except for very specific cases, increasing 'concurrency' through phantoms on a single machine is generally ill-advised, resulting in longer processing times, higher average system loads and worse yet, greater system complexity (and hence ways for things to break). As mentioned earlier, more system-level architectural changes (such as multiple machines, or at least, files storage on different disks/spindles for each process) are required if you want benefit from this sort of work. -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: Monday, October 01, 2012 4:47 PM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe OK - I was trying to create a 'smoother use' of the disk and 'read ahead' -- this example the disk would be chattering from the heads moving all over the place. I was trying to find a way to make this process more 'orderly' -- is there one? -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben Sent: Monday, October 01, 2012 4:48 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe Create an index on a dict pointing at the first character of the key, and have each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9) -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: October-01-12 2:43 PM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe So how would a user 'chop up' a file for parallel processing? Ideally, if here was a Mod 10001 file (or whatever) it would seem like it would be 'ideal' to assign 2000 groups to 5 phantoms -- but I don't know how 'start a BASIC select at Group 2001 or 4001' ... -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen Sent: Monday, October 01, 2012 3:29 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe 0001: OPENSEQ "/tmp/pipetest" TO F.PIPE ELSE STOP "NO PIPE" 0002: LOOP 0003: READSEQ LINE FROM F.PIPE ELSE CONTINUE 0004: PRINT LINE 0005: REPEAT 0006: STOP 0007: END Although, not sure if you might need to sleep a litte between the READSEQ's ELSE and CONTINUE Might suck up cpu time when nothing is writing to the file. Then you could setup a printer in UV that did a "cat - >> /tmp/pipetest" Now your phantom just needs to print to that printer. George -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen Sent: Monday, October 01, 2012 4:16 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe The only thing about a pipe is that once it's closed, I believe it has to be re-opened by both Ends again. So if point a opens one end, and point b opens the other end, once either end closes, It closes for both sides, and both sides would have to reopen again to use. To eliminate this, you could have one end open a file, and have the other sides do a ">>" append To that file - just make sure you include some kind of dataheader so the reading side knows which Process just wrote the data. -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of u2ug Sent: Monday, October 01, 2012 4:11 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe pipes -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Monday, October 01, 2012 4:05 PM To: u2-users@listserver.u2ug.org Subject: [U2] [u2] Parallel processing in Universe What's the largest dataset in the Universe user world? In terms of number of records. I'm wondering if we have any potential for utilities that map-reduce. I suppose you would spawn phantoms but how do they communicate back to the master node? _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
