Hi all, First, thanks to you, the MCE distribution should be passing now with Perl 5.17.0/1/2/3/4/5 as well as Cygwin.
cpan -t M/MA/MARIOROY/MCE-1.103.tar.gz I'm forever thankful to David Cantrell who has allowed me access to his system via a guest account in order to reproduce an issue I wasn't able to reproduce on my end. I've learned a lot during this process. I appreciate what you are doing to help us. Please remove MCE from any blacklist if previously added. I know to take extra measures with MCE before submitting to CPAN. MCE is a new module I created to help folks maximize additional performance by utilizing 2 or up to all available cores on the box using a pool of workers. I remember when I first tried MCE with Perl on Cygwin and Windows (ActiveState & Strawberry Perl). That was a while back. OMG, it wasn't easy. I also remember the times when I tested MCE under FreeBSD and realized that there were no threads or threads::shared. MCE supports forking (no threads required), threads/threads::shared, and forks/forks::shared. After all the stability, I looked for things in which MCE can help parallelize. I came across the Wide Finder project by Tim Bray. The initial results I obtain with MCE were not good enough for me which led to a new option "use_slurpio => 1". MCE is able to run as fast as MMAP IO. I didn't stop there however. I saw that having 16 workers reading IO was actually causing a negative impact to performance. This is due to disk thrashing (becomes random IO when multiple workers are reading simultaneously). So, I then applied the bank-queuing model when reading input data. There's an image showing the bank-queuing model with chunking on the main page at http://code.google.com/p/many-core-engine-perl/. At work, I have tested Net::SNMP + MCE, also Net::SNMP + AnyEvent::SNMP + MCE. It's amazingly crazy to see a 24 way box quickly reach 2300+% processor utilization on the box. It doesn't last long because the process completes very fast when obtaining metrics from 20,000 devices via SNMP. The user_begin, user_end came from thinking of both AWK's and Perl's BEGIN/END blocks. For a long time, I maintained 2 code paths with MCE. One path had a socket pair (one per worker). The other had 4 socket pairs shared by all workers. What I wanted to achieve was being able to preserve output order as if processing serially. In the end, I went with the 4 shared sockets. Eventually, processors will reach 100 cores and I couldn't imagine creating 100 sockets. The cat.pl example included with MCE shows how output order can be preserved. Also included are the the parallel egrep.pl example plus implementation of the Tim Bray Wide Finder project using MCE. Thank you so much for all the testing efforts. Best Regards, Mario BTW: MCE eats really big log files for breakfast :).
