I have heard people have new phrase table formats. The OnDiskPt format is a file accessed with file APIs, not memory mapping. Functionally, it uses the disk cache as shared memory (and the kernel shares the disk cache across processes). There is also some funny accounting going on because a process that depends on the disk cache is not charged for usage of that memory while a mmapped process would be. That means you can run Moses, it looks like it's fitting in virtual memory, and still thrash the disk because you also need enough disk cache to fit the entire phrase table. In this case, it is very slow despite the name OnDiskPt.
Kenneth On 05/25/2012 10:57 AM, Lane Schwartz wrote: > Is there no current option to allow memory mapped phrase tables? I > thought that's what the binary phrase table was. > > Lane > > > On Fri, May 25, 2012 at 10:50 AM, Kenneth Heafield<mo...@kheafield.com> > wrote: >> Use memory mapping (KenLM 8 or 9 on Linux, 9 on non-Linux, or IRSTLM >> with .mm) and the kernel takes care of shared memory for you. >> >> But there is merit to your argument e.g. different weights with the same >> phrase tables. Perhaps the answer is to make the phrase tables memory >> mapped. . . >> >> Kenneth >> >> On 05/25/2012 09:13 AM, Lane Schwartz wrote: >>> I could imagine if you were translating N languages, all into a common >>> target language, that it might be a memory footprint savings to be able >>> to do this all within a common process. The savings would be from being >>> able to have a single language model instance. >>> >>> Lane >>> >>> On Fri, May 25, 2012 at 2:00 AM, Philipp Koehn<pko...@inf.ed.ac.uk >>> <mailto:pko...@inf.ed.ac.uk>> wrote: >>> >>> Hi, >>> >>> my understanding is that this is not currently possible. >>> >>> But why would you want to do this? If you translate with different >>> systems, why not just run different processes? >>> >>> The motivation to do this in the server process is that it avoids >>> keeping multiple server processes at the same time, which is not >>> a concern with batch Moses. >>> >>> -phi >>> >>> On Thu, May 24, 2012 at 12:55 AM, Fong Po Po >>> <fongpui...@yahoo.com.hk<mailto:fongpui...@yahoo.com.hk>> wrote: >>> >>> Dear all: >>> I have read page in >>> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc22 >>> This page say that Moses Server can run in multi >>> translation systems. >>> Can Traditional Moses (not Moses Server) also run in >>> multi translation systems? >>> Can you help me? Thanks! >>> Best Regards, >>> Fong Pui Chi >>> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu<mailto:Moses-support@mit.edu> >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu<mailto:Moses-support@mit.edu> >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >>> >>> >>> -- >>> When a place gets crowded enough to require ID's, social collapse is not >>> far away. It is time to go elsewhere. The best thing about space travel >>> is that it made it possible to go elsewhere. >>> -- R.A. Heinlein, "Time Enough For Love" >>> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> _______________________________________________ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
