I'm very strongly in favor of using boost's filtering streams. See here: https://github.com/moses-smt/mosesdecoder/blob/master/moses/TranslationModel/UG/generic/file_io/ug_stream.cpp
- Uli On Sun, May 3, 2015 at 12:08 PM, Kenneth Heafield <[email protected]> wrote: > http://www.boost.org/doc/libs/1_57_0/libs/iostreams/doc/classes/gzip.html > (best > to scroll to the bottom first). > > On 05/03/2015 04:08 AM, Jeroen Vermeulen wrote: > > On May 2, 2015 2:58:08 AM GMT+07:00, Kenneth Heafield < > [email protected]> wrote: > >> If this comment accurate that gzfilebuf is only used for writing? > >> > >> /** wrapper around gzip input stream. Unknown parentage > >> * @todo replace with boost version - output stream already uses it > >> */ > >> > >> If so I'll just extend util/fake_ofstream.hh to have gzip support. > >> > >> Time to print a bunch of integers: > >> > >> FakeOFStream: > >> > >> real 0m3.460s > >> user 0m3.459s > >> sys 0m0.004s > >> > >> std::cout > >> > >> real 0m23.010s > >> user 0m22.895s > >> sys 0m0.134s > >> > >> Time to print a bunch of floats: > >> > >> FakeOFStream: > >> > >> real 0m34.871s > >> user 0m34.894s > >> sys 0m0.006s > >> > >> std::cout > >> > >> real 1m56.628s > >> user 1m56.690s > >> sys 0m0.037s > >> > >> The conversion is done by https://github.com/miloyip/itoa-benchmark/ > >> and > >> Google double conversion. > >> > >> Kenneth > >> > >> On 05/01/15 14:37, Barry Haddow wrote: > >>> What about the util directory? > >>> > >>> On 1 May 2015 19:13:26 BST, Hieu Hoang <[email protected]> wrote: > >>> > >>> i suppose everything should reference the moses lib. > >>> > >>> that's getting a bit bloated, one day we should look at splitting > >> it up > >>> > >>> On 30/04/2015 10:24, Jeroen Vermeulen wrote: > >>> > >>> Any chance we could re-unify the gzfilebuf and > >> InputFileStream > >>> modules? > >>> Looks like we're carrying around 4 copies of each, and > >> they're > >>> starting > >>> to diverge. > >>> > >>> I'd be happy to make the change, if we know a good reusable > >>> place to put it. > >>> > >>> > >>> Jeroen > >>> > >> ------------------------------------------------------------------------ > >>> > >>> Moses-support mailing list > >>> [email protected] > >>> http://mailman.mit.edu/mailman/listinfo/moses-support > >>> > >>> > >>> > >>> -- > >>> Sent from my Android device with K-9 Mail. Please excuse my brevity. > >>> > >>> > >>> The University of Edinburgh is a charitable body, registered in > >>> Scotland, with registration number SC005336. > >>> > >>> > >>> > >>> _______________________________________________ > >>> Moses-support mailing list > >>> [email protected] > >>> http://mailman.mit.edu/mailman/listinfo/moses-support > >>> > >> _______________________________________________ > >> Moses-support mailing list > >> [email protected] > >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > > I don't have the code at hand but I'm fairly sure I saw it > being used for reading. > > > > Maybe Boost has an equivalent that we can drop in? > > > > > > Jeroen > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
