Personally, I would could "The sees the process as a small-memory process and won't be tempted to kill it when it's running out of memory" as a disadvantage rather than an advantage. If the OOM killer is trying to stabilize the system, this will potentially prevent it from doing so.
The other disadvantage is the lack of accountability. If one is trying to figure out *why* Moses is going so slowly and sees that it's not using up much vmem or memory, this would usually lead to the conclusion that it's not a memory-related issue and that a solution such as buying more RAM won't help. This hidden memory requirement placed on the disk cache can be quite confusing there. -Jon On Sat, May 26, 2012 at 9:09 AM, Hieu Hoang <fishandfrol...@gmail.com>wrote: > You'll also get thrashing with memory-mapped files if you don't have > enough memory. > > Advantage of the file API: > 1. can access 2+GB files even running a 32 bit OS > 2. OS portable > 3. The sees the process as a small-memory process and won't be > tempted to kill it when it's running out of memory > Disadvantage: > 1. Slower (by how much?) > > On 25/05/2012 16:16, Kenneth Heafield wrote: > > I have heard people have new phrase table formats. > > > > The OnDiskPt format is a file accessed with file APIs, not memory > > mapping. Functionally, it uses the disk cache as shared memory (and the > > kernel shares the disk cache across processes). There is also some > > funny accounting going on because a process that depends on the disk > > cache is not charged for usage of that memory while a mmapped process > > would be. That means you can run Moses, it looks like it's fitting in > > virtual memory, and still thrash the disk because you also need enough > > disk cache to fit the entire phrase table. In this case, it is very > > slow despite the name OnDiskPt. > > > > Kenneth > > > > On 05/25/2012 10:57 AM, Lane Schwartz wrote: > >> Is there no current option to allow memory mapped phrase tables? I > >> thought that's what the binary phrase table was. > >> > >> Lane > >> > >> > >> On Fri, May 25, 2012 at 10:50 AM, Kenneth Heafield<mo...@kheafield.com> > wrote: > >>> Use memory mapping (KenLM 8 or 9 on Linux, 9 on non-Linux, or IRSTLM > >>> with .mm) and the kernel takes care of shared memory for you. > >>> > >>> But there is merit to your argument e.g. different weights with the > same > >>> phrase tables. Perhaps the answer is to make the phrase tables memory > >>> mapped. . . > >>> > >>> Kenneth > >>> > >>> On 05/25/2012 09:13 AM, Lane Schwartz wrote: > >>>> I could imagine if you were translating N languages, all into a common > >>>> target language, that it might be a memory footprint savings to be > able > >>>> to do this all within a common process. The savings would be from > being > >>>> able to have a single language model instance. > >>>> > >>>> Lane > >>>> > >>>> On Fri, May 25, 2012 at 2:00 AM, Philipp Koehn<pko...@inf.ed.ac.uk > >>>> <mailto:pko...@inf.ed.ac.uk>> wrote: > >>>> > >>>> Hi, > >>>> > >>>> my understanding is that this is not currently possible. > >>>> > >>>> But why would you want to do this? If you translate with > different > >>>> systems, why not just run different processes? > >>>> > >>>> The motivation to do this in the server process is that it > avoids > >>>> keeping multiple server processes at the same time, which is not > >>>> a concern with batch Moses. > >>>> > >>>> -phi > >>>> > >>>> On Thu, May 24, 2012 at 12:55 AM, Fong Po Po > >>>> <fongpui...@yahoo.com.hk<mailto:fongpui...@yahoo.com.hk>> > wrote: > >>>> > >>>> Dear all: > >>>> I have read page in > >>>> > http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc22 > >>>> This page say that Moses Server can run in multi > >>>> translation systems. > >>>> Can Traditional Moses (not Moses Server) also run in > >>>> multi translation systems? > >>>> Can you help me? Thanks! > >>>> Best Regards, > >>>> Fong Pui Chi > >>>> > >>>> > >>>> _______________________________________________ > >>>> Moses-support mailing list > >>>> Moses-support@mit.edu<mailto:Moses-support@mit.edu> > >>>> http://mailman.mit.edu/mailman/listinfo/moses-support > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> Moses-support mailing list > >>>> Moses-support@mit.edu<mailto:Moses-support@mit.edu> > >>>> http://mailman.mit.edu/mailman/listinfo/moses-support > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> When a place gets crowded enough to require ID's, social collapse is > not > >>>> far away. It is time to go elsewhere. The best thing about space > travel > >>>> is that it made it possible to go elsewhere. > >>>> -- R.A. Heinlein, "Time Enough For Love" > >>>> > >>>> > >>>> _______________________________________________ > >>>> Moses-support mailing list > >>>> Moses-support@mit.edu > >>>> http://mailman.mit.edu/mailman/listinfo/moses-support > >>> _______________________________________________ > >>> Moses-support mailing list > >>> Moses-support@mit.edu > >>> http://mailman.mit.edu/mailman/listinfo/moses-support > >> > >> > > _______________________________________________ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
