[Zeek-Dev] Re: netmap 2.0.0 vs. zeek 4.0.0
Btw, there's work underway to simplify building external plugins directly into Zeek: https://github.com/zeek/zeek/pull/1416 That might be another option in the not-too-distant future. Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: netmap 2.0.0 vs. zeek 4.0.0
On 3/16/21 2:10 PM, Craig Leres wrote: It's easy enough to put zeek-config on PATH but it doesn't work because it is configured/built to install in /usr/local but during the packaging process it's actually installed in /wrkdirs/usr/ports/security/zeek/work/stage/usr/local. This causes zeek-config to report the target install dir, not the current actual dir: Ah, I see, thank you ... this sounds a lot like DESTDIR. Something would have to redirect the plugin build and install to that workdir for purposes of the plugin build. zeek-config doesn't currently support that. A few thoughts: - Going back to your first message, I'm not sure why CXXFLAGS didn't work for you for locating headers -- I just tried a CXXFLAGS=-I/path/to/dummy/include ./configure ... and I can see that directory appear in the build output (via cmake --build build -v). So perhaps the header inclusion actually worked, but you hit other trouble later on? - You could try patching the generated zeek-config so it prefixes paths with the workdir, and undo that for the version that eventually gets installed. This might be something we need to add to it for this scenario. The paths it reports are all defined right at the top of the script, so it wouldn't be hard to trick it to use the workdir. I don't know if that's enough, though. - I'm coming back to --zeek-dist now, which should find the required headers. I just tried a build with it and it worked for me ... are you able to point it at the built source tree? I'd suspect it's different from the staged installation. I don't understand how --with-netmap works, looking in cmake/FindNetmap.cmake, I think it wants netmap_user.h which is present in /usr/include... But it doesn't find it? This might be a detail about the tree layout -- for me, when I say --with-netmap=/foo/bar, it locates netmap_user.h in /foo/bar/include/net/netmap_user.h. I'm afraid if none of the above helps, you'll need input from somebody who actually uses FreeBSD -- I don't. Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: netmap 2.0.0 vs. zeek 4.0.0
Hey Craig,
It's been a while! :)
On 3/16/21 10:54 AM, Craig Leres wrote:
Due to the way FreeBSD packages are built at this point zeek is
installed but in /wrkdirs/usr/ports/security/zeek/work/stage/usr/local,
not /usr/local.
As long as the configure/build process is able to find zeek-config, the
helper script that reports paths for a given Zeek installation, it
shouldn't matter where you have Zeek installed. Is
/wrkdirs/usr/ports/security/zeek/work/stage/usr/local/bin in your PATH?
zeek-config --include_dir should report the right include trees then.
post-install-NETMAP-on:
${LN} -s ../.build ${WRKSRC}/build
cd ${WRKSRC_zeek_netmap} && ./configure --zeek-dist=${WRKSRC} \
--install-root=${STAGEDIR}${PREFIX}/lib/zeek/plugins
cd ${WRKSRC_zeek_netmap}/build && make && make install
This is an aside, but I'd advise against using --zeek-dist unless you
have to build without an installed Zeek tree. While it should work,
source trees shouldn't be required these days to get a build. If you
have Zeek installed and zeek-config in your path, I'd hope that a plain
./configure && cd build && make
will work. If you also need to point at an alternative location for the
Netmap headers, try --with-netmap=.
Let us know how it goes...
Best,
Christian
___
zeek-dev mailing list -- zeek-dev@lists.zeek.org
To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Proposed change to lambda semantics - shallow copying rather than references
On 12/11/20 1:47 AM, Robin Sommer wrote:
Sounds like a way forward then to both address the current concern,
and improve this overall. Does this work for everybody?
Yes for me, just one comment: I'm a bit nervous about getting too
inspired by C++ syntax. With every new standard round it's looking more
like control character soup. Vern, I'm not sure what you had in mind
here ... but perhaps instead of something very close to C++, like
local f = function[=]() { print r$i; };
to capture that I'd like r to be a deep copy, maybe we could consider
something more Zeek-style, perhaps via attributes:
local f = function() { print r$i; };
for all variables or
local f = function() =r { print r$i; };
just for r, with shallow-copy the default. More typing, yes, but
arguably less cryptic, and this is likely a relatively rarely used feature.
Best,
Christian
___
zeek-dev mailing list -- zeek-dev@lists.zeek.org
To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Proposed change to lambda semantics - shallow copying rather than references
On 12/9/20 2:28 PM, Vern Paxson wrote: It already varies with type of variable for assignments and function call parameters. Thus I’m puzzled at the desire for deep-copy over shallow-copy, given that Zeek is already primarily shallow-copy. Access to the variable here is so natural (in the script code, not the implementation) that I don't immediately think of assignment semantics at all. So I intuitively compared it to other languages, where I'd tend to expect i to be a (deep) reference. If we cover the behavior in the docs I'm fine with either. Regardless of where this lands, your point strikes me as a great one to work into our current docs push. I just did a bit of digging and I can't find much that describes Zeek's shallow/deep or reference/value semantics. I see a bit in the description of copy(), in that of closures, and in CHANGES by some guy in 2005: - The manual has been updated to clarify that aggregate values in events are passed as shallow copies, so that modifications to elements of the values after posting the event but before it's handled will be visible to the handlers for the events (Christian Kreibich). :) Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Proposed change to lambda semantics - shallow copying rather than references
On 12/9/20 11:10 AM, Vern Paxson wrote: For the script optimization/compilation work I’ve been doing, I’ve been looking into what it will take to compile lambdas (anonymous functions). I think the current functionality would be better, since it's how I'd expect it to behave (which probably just reflects my bias toward procedural languages). That said, since Zeek only gained support for closures relatively recently there's likely little precedent, and there's a way out via records. So I'd be okay with it. It'd be good to explore whether Zeek could warn if the change affects existing code. Not sure how feasible that is... Hth, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: [Zeek-Def] Re: Platform support policy
On 11/9/20 1:41 AM, Robin Sommer wrote: Yeah, agree, that sounds better than maintaining the information separately. Cool. I've updated the wiki page to refer to our CI configs for the details. I'm definitely not the expert here but it all looks like Catalina with varying additions: Yeah, I saw that, but not sure if that means they are actively removing older images. I'll see if I can find out. Hah, I asked them too and they actually mentioned their email reply to you! https://github.com/cirruslabs/cirrus-ci-docs/issues/726 So it looks like we can start opening up the version window soon. Yay. I've put that in the wiki page too. Here's the PR for the corresponding updates, now out of draft state: https://github.com/zeek/zeek/pull/1268 Thanks! Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: [Zeek-Def] Re: Platform support policy
On 11/6/20 2:03 AM, Robin Sommer wrote: - Which of these come with additional requirements beyond just OS base packages? We should note those. What I can think of: - CentOS: We require devtoolset (which version?) and probably EPEL? I don't recall how people get some of the dependencies in place (bison, flex, cmake). Mhmm ... this seems a bit tricky. At least in our CI setups each platform requires some stuff to be installed to allow a build, so I'm wondering whether instead of spelling them out in the wiki page we could just point at our CI? That'd be Dockerfiles for anything Linux and prepare.sh for FreeBSD and macOS. The benefit would be that we'd maintain this in one place only. We could invest a bit of time in documenting the Dockerfiles/prepare.sh scripts so they explain these? - macOS: Homebrew I suppose. Yeah, worth calling out because Cirrus includes it in their image, so you won't notice it when looking purely at only our CI setups. - On CentOS, if we limit support to "full updates", CentOS 7 will drop off soon. I'm wondering if we should use versions with maintenance updates instead? (I'm glad CentOS 6 just went out of life support :-) :) -- yes, I mislabeled that one. I actually meant to say "Maintenance Updates". Fixed in the wiki page. - Does anybody know if Cirrus keeps previous macOS images available? If so, I'd suggest we keep the two most recent macOS versions. I'm definitely not the expert here but it all looks like Catalina with varying additions: https://cirrus-ci.org/guide/macOS/ Btw I didn't include anything about architectures ... for Debian 9 we currently have a 32-bit container, for some other platforms those are still available too. Do we still care about 32-bit? Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Platform support policy
On 11/5/20 8:51 PM, Christian Kreibich wrote: I'm putting together a PR (still a draft) for bringing our CI in line with what's in that page: Ahem -- that was supposed to be: https://github.com/zeek/zeek/pull/1268/ Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Platform support policy
Hi folks, Sorry for the delay here -- I've now put together a page: https://github.com/zeek/zeek/wiki/Platform-Support-Policy I've also added an entry to the calendar for macOS (Catalina, using the availability of images in Cirrus CI as a driver), and I'm putting together a PR (still a draft) for bringing our CI in line with what's in that page: https://github.com/zeek/zeek/wiki/Platform-Support-Policy Let me know your thoughts... Thanks, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Platform support policy
On 10/22/20 1:40 AM, Robin Sommer wrote: Want to start a Wiki page where we collect the pieces for our new policy? Can link to those calendars from there. Sure thing. I agree with your remaining comments, too. Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Platform support policy
On 10/5/20 4:08 AM, Robin Sommer wrote: The advantage of this is that we won’t rely on having to specify specific compiler versions, which isn’t well-defined (because things also depend on libstdc++ version, system libs, etc.), but instead on whether people can actually compile Zeek on the platforms they are using. A disadvantage is that we’ll need to spend the time to ensure our CI setup keeps matching the policies, and gets updated as distribution updates come out. I like this a lot! At the end of the day, what matters for our builds is what the distros ship, and this covers that. Plus, it's not that hard to update the CI images to keep them current. We could put together a release calendar to remind us when it's time to update CI. I took a quick look and couldn't immediately find one. I'd be happy to put one together. Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Unit tests in plugins?
On 9/9/20 9:56 AM, Seth Hall wrote: Has anyone put any thought to how we could create and run unit tests in Zeek plugins? I don't think any work has been done on that yet, but I'd love to be able to create unit tests in plugins. Yeah I've tinkered a bit with this. I was able to get a unit test to run out of an plugin in a loaded Zeek package (for current master) by doing two things: - Include doctest.h in the plugin source tree. - Move the doctest execution in src/zeek-setup.cc down until after the plugin registration. With this, zeek --test -ltc shows the test case, and it also runs: == /home/christian/.zkg/clones/package/utest/src/Plugin.cc:21: TEST CASE: plugin doctest /home/christian/.zkg/clones/package/utest/src/Plugin.cc:23: SUCCESS: CHECK( true ) is correct! values: CHECK( true ) But that's obviously a hacky setup. To do this properly, we should add doctest.h to the set of things that Zeek installs (so plugins don't have to ship their own), and we'll have to rework a bit of the setup logic. And, I don't know how/whether this will work with other compilers/platforms; this was just my local system. Are you up for creating an issue for this one, Seth? Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Moving policy scripts into packages
Great summary, thanks Robin. On 8/31/20 2:14 AM, Robin Sommer wrote: - zkg: - Could we add a way to "prime" zkg's package cache so that a Zeek distribution could distribute a snapshot of "zeek-packages" for direct use; but zkg would still pull in updates if online access is available? Related to this, I was wondering about zkg's status as an affiliated project ... if we strengthen the notion of packages from the core distribution, we may want to ensure zkg can be available from the outset (as a core component)? I tried to look at some equivalents in other environments ... for example, it looks like when you install Python/Ruby from the official tarballs you get pip/gem out of the box. Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Moving policy scripts into packages
On 8/25/20 1:27 AM, Jan Grashöfer wrote: I think the underlying issue extends to the pcaps as well. If I am not mistaken, a significant number of test cases make use of the same pcaps. Having multiple copies of them scattered across different repositories doesn't feel right. Good point, yeah. I took a look at this. The btests in the Zeek distribution use 352 different pcaps. The tests for the policy folder use 30; 18 of them are also used by other tests. That set is small, less than 1MB. Use of the same pcap across the policy tests is pretty limited and would likely occur only within one new package. So it's not too bad -- if those tests migrate into new Zeek packages, we would duplicate those 18 pcaps, but mostly just once. Other Zeek packages seem pretty unlikely to need a pcap from the Zeek distribution, but I'd argue they probably benefit from a standard random.seed and some of the canonicalizers. An alternative would be to install the full set of pcaps (that's 23MB), probably optionally, and make the availability of the pcap a test requirement. What do you think? I personally could live with duplicating those pcaps. Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Moving policy scripts into packages
On 8/24/20 11:49 AM, Johanna Amann via zeek-dev wrote: * Testing: Currently, some of the policy scripts have tests that use Zeek functionality in rather unique ways / or are the only tests for some Zeek functionality. The SSL validation scripts are one example. This, from my point of view, it would be neat to have a way to still easily install a rather large set of packages (potentially nearly everything that is in policy at the moment) and run test on them. On a related note, I think it would be quite helpful if the Zeek install tree would include some of the standard btest helpers, such as random.seed and the various canonification scripts. This also comes with a fun problem. Sometimes we perform changes to Zeek that change a lot of the test baselines - especially when we touch something that affects connection-ID hashing, or the order of elements in hashmaps. These cases might now require an update to the test-cases in a large number of packages. It would be neat to have an easy way to perform this. I agree that this would be really handy. We could build some tooling that would still let you maintain the packages as individual repos but that enables such use use cases. It'd be pretty handy to have a zkg command that clones a given package plus all of its dependencies for development work, for example. Since zkg already speaks git rather well, it could even synthesize a submodule structure. Thanks, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
[Zeek-Dev] Re: Moving policy scripts into packages
On 8/24/20 9:51 AM, Robin Sommer wrote: Also, one additional thought: Jon reminded me that zkg can manage dependencies already. So the "collection" I mentioned could be a meta-package that depends on all the ones we want. Yeah, agreed -- I prefer #2 for the same reason. Best, Christian ___ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
