On Fri, Jun 8, 2018 at 10:23 AM Robin Sommer <ro...@icir.org> wrote: > > > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > > Known::use_host_store=F Known::use_service_store=F > > Known::use_cert_store=F > > That indeed gets it way down, though still not back to the same level > on my box: > > 170.49user 58.14system 1:55.94elapsed 197%CPU > > (pre-master: 73.72user 7.90system 1:06.53elapsed 122%CPU)
Just double-checking: same configure/build flags were used between the two? Here's the more precise results I got from running on a macOS and Linux system: # Linux master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup real 0m32.572s user 0m40.926s sys 0m7.873s # Linux master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup Known::use_host_store=F Known::use_cert_store=F Known::use_service_store=F real 0m25.603s user 0m23.311s sys 0m7.952s # Linux pre-broker (7a6f502) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup real 0m24.785s user 0m21.230s sys 0m8.341s # macOS master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup.bro real 0m38.775s user 0m42.365s sys 0m8.950s # macOS master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup.bro Known::use_host_store=F Known::use_cert_store=F Known::use_service_store=F real 0m32.975s user 0m33.774s sys 0m4.409s # macOS pre-broker (7a6f502) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup.bro real 0m30.785s user 0m31.840s sys 0m3.788s > Are there more places where Bro's waiting for Broker in pcap mode? Not that I can think of. > Yeah, I remember that discussion. It's the trade-off between > performance and consistency. Where's the code that's doing this > blocking? I just know that Manager::AdvanceTime() and also Manager::TrackStoreQuery() -> FlushPendingQueries() can block/spin waiting on actor/threading in pcap mode. > Would it be possible to not block right away, but sync up > with Broker when events are flushed the next time? (Like we had > discussed before as a general mechanism for making async operations > consistent) I think the way to make an async operation most consistent is to model it as a synchronous operation? That's what I'm doing now at least, and if I try moving FlushPendingQueries() to later (before the AdvanceTime() call) in an attempt to possibly have more queries in the queue/buffer, I get the external test suites failing. At least on my own test systems, I don't have much performance to recover by going this route anyway, so maybe we need to dig more into why your results are different. Only thing I'm thinking is that your test system maybe does a poorer job of scheduling the right thread to run in order for the FlushPendingQueries() spin-loop to actually finish. - Jon _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev