Just posted this to the pick/multivalue Google group, but figure there may be
some (larger?) U2 only people who may have valuable insight on the subject, so
We have been using TeamViewer (www.teamviewer.com) for the past 18 months or
so, and I'm generally very happy with it - I can access
On Tue, Oct 2, 2012 at 5:58 PM, Symeon Breen wrote:
> However map reduce and hadoop are pretty horrible things. Even Google have
> moved away from it with Caffiene etc.
>
Going OT a little, i think Google is replacing "BigTable" which was part of
Caffeine in 2010 with "Spanner" now. Here is a d
Depends on what you call a "no brainer" --> to me, $4K for an 800Mb Intel 910
SSD seems "reasonable" for what you get (10x full drive writes every day for 5
years has the endurance angle covered IMHO - 400Gb is $2K if your database will
fit) and by todays standards represents "reasonable" v
Could also avoid the lock contention if each phantom had knowledge of the
others, so "phantom 1" could only process @ID 1, 6, 11 etc., phantom 2 would do
2,7,12 & so on
Of course, if you are operating with a select list, this already implies that
you have processed the file once, so your "batch
You may not need to know what *group* you are in per se, if you are willing to
use the file stats record.
You can determine from the last stats, how many records are in your file.
Then your master program just reads the keys until it gets to the 50,000th key
(or whatever), and then spawns a pha
Which was my question -- was there a way to 'jump to' a group or 'BASIC
SELECT' with a'starting/ending' group -- so that again, 10001 moduo, one
phantom does 'groups' 1-2000, next phantom does 'groups' 2001-4000 etc...
But can't see that it's really possible without jumping through hoops that
make
OK. I See what your saying...I'll buy that.
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent: Tuesday, October 02, 2012 1:42 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] [u2] Parallel processing
The point of the caching concern is related to the read ahead, and you will
still get some benefit from this, if your five phantoms are reading their
*portion* of the file in order, which they should.
-Original Message-
From: George Gallen
To: U2 Users List
Sent: Tue, Oct 2, 2012 10
If 5 phantoms were running, and read in order but from 5 different starting
points, the records would
Essentially still be processed in a random order, if you were to layout the
record ID's as they get
Processed.
George
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mai
The idea of the phantoms would be to read the file in order, not randomly, just
inorder from five different starting points.
So you should still get the benefit of some caching.
-Original Message-
From: Daniel McGrath
To: U2 Users List
Sent: Tue, Oct 2, 2012 10:32 am
Subject: Re: [U2
Yes, SSD will definitely help. Just keep in mind, it doesn't prevent all
negatives in regards to I/O, particularly with regards to caching.
Disk caching in a modern system is fairly complex, but at the high level it is
not only done by the controller, but by the OS as well. So randomly flying
a
Great point!! I think we can agree that 'spinning media latency' is the
enemy and having phantoms increasing the 'head dance' can make things worse,
not better!
Many problems go away or become trivial as the spinning media trails to the
sunset. I've advised customers that just moving 'code fil
You've highlighted one problem here.
By having multiple processes accessing the disk in different locations, you
destroy cache optimization and seek times. More phantoms = less performance.
This assumes I/O is a bigger concern than CPU, which is generally the case.
More phantoms = more communic
AH - would not even have to 'delete' as long as the 'locks' are held long
enough -- meaning if you know you will have 20 phantoms, each phantom would
keep a list of 'keys locked' and once it hits 21 (or 40 if you want
insurance LOL) in the list, would unlock earliest lock -- that way there is
no wa
What if you created a duplicate file, did a SELECT and saved the list
(non-sorted).
Each of the phantoms would do a getlist and loop through using readlist/readu
and if the record were already locked, skip it until it reads An unlocked
record
(and locks it). Delete the record when finished.
In my example, I would grab 'whatever' records were hashed in the to 'group'
-- while it's not perfect since there are 'overflow' - was just trying to
think of a way to break a file into pieces that would otherwise process much
like a BASIC select - just grab the 'group' and go I can see it's
Yes the low numbers are used more often.
However if you have sequential keys, just use the *last* two digits instead of
the first two
-Original Message-
From: Wols Lists
To: u2-users
Sent: Tue, Oct 2, 2012 1:17 am
Subject: Re: [U2] [u2] Parallel processing in Universe
On 01/10/12 2
Only outside of U2 using UniObjects can you achieve any type of parallel
activity. We have through UniObjects got 80 processes working from a single
Eclipse session through the use of threads in Java.
UniObjects creates individual uvapi_slave or udapi_slave for each of these
processes but the syst
On 02/10/12 15:28, George Gallen wrote:
> What about an striped array of SSD with a backup battery to flush the write
> buffer on power fail.
> No more dangerous (IMO) than an array of hard drives - but given the limited
> write times of an SSD
> That could be more of a danger, unless your using
Only outside of U2 using UniObjects can you achieve any type of parallel
activity. We have through UniObjects got 80 processes working from a single
Eclipse session through the use of threads in Java.
UniObjects creates individual uvapi_slave or udapi_slave for each of these
processes but the syst
What about an striped array of SSD with a backup battery to flush the write
buffer on power fail.
No more dangerous (IMO) than an array of hard drives - but given the limited
write times of an SSD
That could be more of a danger, unless your using larger drives and not a lot
of data so the drive
On 02/10/12 03:49, Ross Ferris wrote:
> If the file were big enough, and already had part files, then I believe that
> you could have a phantom process each of the individual parts. Failing that,
> get an SSD relatively cheap, and will give your processing a reasonable
> kick along!!
>
Jus
On 01/10/12 22:47, Robert Houben wrote:
> Create an index on a dict pointing at the first character of the key, and
> have each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9)
>
Actually, this is a very BAD way of chopping up a file into five even
chunks.
I'm not sure of the stats, but on any
On 02/10/12 08:23, Hona, David wrote:
> The installation instructions of Rocket is quite good and does indeed mention
> the need to use "cpio" on UNIX servers. See "Quick Installation" and
> "Step-by-step Instructions" (of NEWINSTALL.PDF)...
>
> However, the instructions from Rocket could be imp
Oracle and sql server both use map reduce internally when doing collations
and totals. However they work differently to U2 in that they have one big
process that runs queries from the clients. This process can then cache,
multithread and map reduce. U2 is differently architected in that the client
The installation instructions of Rocket is quite good and does indeed mention
the need to use "cpio" on UNIX servers. See "Quick Installation" and
"Step-by-step Instructions" (of NEWINSTALL.PDF)...
However, the instructions from Rocket could be improved - with a minor
revision, as the Installat
26 matches
Mail list logo
