On 13 Sep 2011, at 22:53, Scott Ribe wrote:
> On Sep 13, 2011, at 3:07 PM, jonat...@mugginsoft.com wrote:
>
>> Are you sure about this?
>
> Yes. From man fork:
>
> There are limits to what you can do in the child process. To be totally
> safe you should restrict yourself to only exec
-only dense matrix). Still, this
should work out, since after the initial mmap load, it's quite speedy.
From: Scott Ribe
To: Steve Sisak
Cc: Jon Sigman ; list-cocoa-dev
Sent: Tuesday, September 13, 2011 2:55 PM
Subject: Re: Task dispatching
On Sep 13, 2011,
On Sep 13, 2011, at 3:28 PM, Steve Sisak wrote:
> Actually, I believe that mmap just maps the the file into memory using the VM
> system -- it's initially paged out and read in as each page is touched...
Yes, thus my comment. Did the OP actually read all values? The time seems too
fast for that
On Sep 13, 2011, at 3:07 PM, jonat...@mugginsoft.com wrote:
> Are you sure about this?
Yes. From man fork:
There are limits to what you can do in the child process. To be totally
safe you should restrict yourself to only executing async-signal safe
operations until such time as
At 2:39 PM -0600 9/13/11, Scott Ribe wrote:
\Note that your disk is not really providing 1.25GB/sec reads. What
was probably happening is that the file contents were in system
cache because of your prior reads. (Assuming that with the mmap code
you were actually reading all the data...)
Actua
On 13 Sep 2011, at 21:34, Scott Ribe wrote:
> On Sep 13, 2011, at 1:50 PM, Stephen J. Butler wrote:
>
>> Programs like postgres run just fine in OS X and they rely heavily on
>> forking.
>
> Of course, but they don't just fork then continue execution of the same
> image, which is what was bei
On Sep 13, 2011, at 2:10 PM, Jon Sigman wrote:
> So, I'm stunned that this would be so much faster than using individual
> fread() calls for each element.
I'm not. fread has a lot of overhead relative to reading 8 bytes. If your data
structure really is just a big non-sparse n-dimension array o
On Sep 13, 2011, at 1:50 PM, Stephen J. Butler wrote:
> Programs like postgres run just fine in OS X and they rely heavily on forking.
Of course, but they don't just fork then continue execution of the same image,
which is what was being suggested. They fork and then exec a new image.
--
Scott
I quickly wrote up a little program to test out mmap()'ing one of the 1GB
files. It works, no kernel changes needed, and it only takes 0.871 secs!
(51.209 secs if Time Machine is running ;) So, I'm stunned that this would be
so much faster than using individual fread() calls for each element. (
On Tue, Sep 13, 2011 at 1:36 PM, Scott Ribe wrote:
> On Sep 13, 2011, at 12:23 PM, Jens Alfke wrote:
>> The forked process is an exact clone of the original, with its address
>> space copy-on-write, so it’s already up and running without any startup time.
>
> Yeah, but about the only thing you c
On Sep 13, 2011, at 1:14 PM, Jens Alfke wrote:
> That’s true in the general case on OS X, but my understanding is that as long
> as you stick to POSIX APIs you can do actual work in the forked process.
No, you have to stick to async-signal safe POSIX APIs, which are a very very
small subset of
On Sep 13, 2011, at 11:36 AM, Scott Ribe wrote:
> Yeah, but about the only thing you can do in that forked process is exec a
> new image over it. This is not a technique for memory sharing.
That’s true in the general case on OS X, but my understanding is that as long
as you stick to POSIX APIs
Jon Sigman wrote:
Also, won't I need to increase shmmax in the kernel, especially if
I have numerous "flavors" of the 1GB matrix to load?
What is a "flavor" of a matrix?
You need to explain what you're doing in terms of the data and its
representation. How is the matrix data represented o
On Sep 13, 2011, at 12:23 PM, Jens Alfke wrote:
> The forked process is an exact clone of the original, with its address space
> copy-on-write, so it’s already up and running without any startup time.
Yeah, but about the only thing you can do in that forked process is exec a new
image over it.
On Sep 13, 2011, at 11:46 AM, Jon Sigman wrote:
> The shared memory approach might work (I assume you mean with shm_open(),
> mmap(), shmget(), shmat(), etc.?) But isn't there a problem with the shm
> mechanism on multiprocessor hardware, or was that from years past? Also,
> won't I need to inc
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 9/13/11 10:30 AM, Scott Ribe wrote:
> - A couple of minutes still seems like a long time to load it.
> Perhaps there is a more efficient way to store & load it?
Perhaps there are also matrix operations that occur as part of the
load process?
Rega
On Sep 13, 2011, at 10:11 AM, Jon Sigman wrote:
> Startup involves loading a >1GB data matrix from disk into memory.
> Transformations are then done using this matrix on the 1MB data blocks that
> are sent in via TCP/IP. That's why I can't simply start new tasks on-demand,
> the startup time i
__
From: Scott Ribe
To: Jon Sigman
Cc: Cocoa- Dev List
Sent: Tuesday, September 13, 2011 10:30 AM
Subject: Re: Task dispatching
On Sep 13, 2011, at 11:11 AM, Jon Sigman wrote:
> Startup involves loading a >1GB data matrix from disk into memory.
> Transformations are
On Sep 13, 2011, at 11:11 AM, Jon Sigman wrote:
> Startup involves loading a >1GB data matrix from disk into memory.
> Transformations are then done using this matrix on the 1MB data blocks that
> are sent in via TCP/IP. That's why I can't simply start new tasks on-demand,
> the startup time is
ting a "server" in Cocoa?
From: Scott Ribe
To: Jon Sigman
Cc: Cocoa- Dev List
Sent: Tuesday, September 13, 2011 9:55 AM
Subject: Re: Task dispatching
On Sep 13, 2011, at 10:42 AM, Jon Sigman wrote:
> Design-wise, should I have a lightweight front-en
On Sep 13, 2011, at 10:42 AM, Jon Sigman wrote:
> Design-wise, should I have a lightweight front-end process that accepts all
> inbound requests and then dispatch the requests to one of the other idling
> processes?
That is an extremely common idiom for servers. Whether you should do that, or
I have a Cocoa/Foundation application running in the background that receives a
1MB block of data, processes it, and sends back a new 1MB block of data using
TCP/IP over the internet. The program takes a couple of minutes to come up and
initialize, so it needs to be up and stay up waiting for pr
22 matches
Mail list logo
