If you're reading the data out of the file from inside a generator (ie - if
load_items returns a generator that accesses the HDF5 file) then as the
worker processes consume the work items the file is actually being opened
and read from a worker thread in the master process.
Regards,
Owen
On 2 November 2012 21:49, Ben Elliston <b...@air.net.au> wrote:
> Hi Francesc
>
> > Hmm, now that I think, Blosc is not thread safe, and that can bring
> > these sort of problems if you use it from several threads (but it
> > should be safe when using several *processes*).
>
> I am using multiprocessing.Pool, like so:
>
> if __name__ == '__main__':
> pool = Pool(processes=2) # start 2 worker processes
> items = load_items ()
> pool.map (process_items, items)
>
> Ben
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQIVAwUBUJQx0+ZqTimv57y9AQigFQ//XWpDoIve2PDag4SG/JBu6Y4D5X8pZcfA
> froFDmju5dGtlCaRKUc/puioFRukyD2s5QCV9hvYIGQ2EYkts1eKLQnYRvcV/37D
> J6QVEGUcuqfdj6lnGZSiDHr24rCeT3oGozbYO0/6casyV4iIuRjOzghWgKjV7ko1
> N/dy2UGK0S1S2Ws/OnkzDlbXiZShHfjLw3au2TtCdXaPcA4X1aMs1qLzEzvd+gJb
> MLHy3MVtGCxjtnB3Vzi/2UmgMfB6hFuGugD2Yp2i0SxFIlTS7cIXYB2beL+x+y4Q
> MtcZZO7QOvTvoExRje0BWR0e0BAOumKACKiU7Uq8L0+rwT6NWPfOVfe6KhieHvi/
> bh8oiNl2tekB+UE5JQ6Yi13YwfReyA1M8RFRsrQ3fXCWaQ6+Hx3m8+t/q4bfDhy4
> wFrC4N3hIkqMNI589aju8vVWerSdKDrzqFjcLBp8zfY718qm0ulGz0bBsKNPzr8y
> 7AN+/5StSos29oBTv3p3s7YDLjXoDi4XwZPR0aMt9pO2JrPxVUda+oi8/1AT46RS
> QoyFGmkqotucR22rWXbv9wZdHm9SsxGZakNpeQYxN9uf2aaqP0eodbgmSPx1ST8P
> 8vU+uGqrCb564KW5eJmQCgNZMXo1uCAqCdF5LZT+h1ncWy9//bv9gW2mityDNs6j
> YOqKMSOj9e0=
> =ZvxS
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> LogMeIn Central: Instant, anywhere, Remote PC access and management.
> Stay in control, update software, and manage PCs from one command center
> Diagnose problems and improve visibility into emerging IT issues
> Automate, monitor and manage. Do more in less time with Central
> http://p.sf.net/sfu/logmein12331_d2d
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
LogMeIn Central: Instant, anywhere, Remote PC access and management.
Stay in control, update software, and manage PCs from one command center
Diagnose problems and improve visibility into emerging IT issues
Automate, monitor and manage. Do more in less time with Central
http://p.sf.net/sfu/logmein12331_d2d
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
