I haven't done any actual benchmarks for the resource usage, however as an admin I would rather not have numerous processes for a single data channel that needs to be parsed and split.
My apologies for sort of hijacking this thread with a cPanel integration. It's just so widely used, that I'm sure others here have had the need or thought at some point and figured I would share my findings. As cPanel servers are known to house hundreds of domains, per domain varnishncsa simply isn't an option, which is why I chose to piggyback on splitlogs as it's already a supported option of theirs. On Nov 13, 2016 17:44, "Guillaume Quintard" <[email protected]> wrote: > Is the resources argument that compelling? I find it cleaner to have one > ncsa per domain, plus the ncsas will read from memory, which is super fast. > > Sure you can then split logs afterwards using awk or whatever, but that's > adding an extra layer and serializing a process that doesn't need to be. > > But I'm not an admin, so I may be off. > > On Nov 13, 2016 16:27, "Andrei" <[email protected]> wrote: > >> Hello, >> >> By not using splitlogs or an intermediary script, you will be forced to >> run multiple instances of varnishncsa, which isn't optimal if you host >> multiple domains. The more traffic/domains you have, the more resources you >> will consume on parsing the same data across multiple channels. >> Yes, varnishncsa supports VSL, however I think your approach is a bit off >> (no offense). As in, you need to shift the log writing process away >> from varnishncsa. By doing so, you only have one instance of varnishncsa >> using resources to gather the data, which is then fed to a parser that >> handles the per domain log splits and writes. That's where 'splitlogs' came >> into play. >> >> As your question does raise some interest in the cPanel community (myself >> and Miguel González on this list for example), I threw together a quick >> Perl script that will in short, pipe and parse data between varnishncsa and >> the splitlogs binary for cache hits. This lets splitlogs handle the queued >> log writes which are later parsed for cPanel bandwidth usage and graphs, >> webalizer, awstats, logaholic, etc - https://github.com/AndreiG6/vscp >> >> >> >> On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced <[email protected]> >> wrote: >> >>> Same Hello here ;) >>> >>> did have a more in depth look in the manual and figured out that >>> varnishncsa does support VSL query. >>> so someone could filter on the Request Header and Host >>> >>> varnishncsa -q "ReqHeader ~ '^Host: .*example.com'" >>> >>> which would produce a log for a specific domain only >>> >>> it then would need multiple varnishncsa instances for logging per >>> domain, which I found here: >>> >>> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/ >>> >>> I use varnish version 5 and then there would be no need for splitlog and >>> the logs would be created directly. >>> >>> please correct me if I'm wrong? >>> >>> thanks for your time & help >>> Becki >>> >>> >>> Am 12.11.2016 um 17:05 schrieb Andrei: >>> >>>> Hello again, >>>> >>>> My apologies for not explaining my thoughts better earlier then. Afaik, >>>> varnishncsa does not have a native method to split output based on >>>> different parameters. The method I was thinking of was based on piping >>>> varnishncsa output through splitlogs (or similar) for the log processing >>>> and writeouts. Since replying earlier, I've got this working on a cPanel >>>> server with piped logging enabled for Apache using the following two for >>>> example (X-Port is a custom header set in vcl_recv related to SSL >>>> offloading, but you can use a static value or similar custom header): >>>> >>>> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b >>>> \"%{Referer}i\" \"%{User-agent}i\""|sed -e >>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname` >>>> --mainout=/usr/local/apache/logs/access_log >>>> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e >>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname` >>>> --suffix=-bytes_log >>>> >>>> The above pipes the requests to the splitlogs binary which queues then >>>> writes to separate logs per domain, that are later processed by the >>>> cPanel log stats apps. Either way, I believe you need an intermediary >>>> script to queue and write the log entries per domain. While looking into >>>> this process, I ran across this little tidbit which you may find of use >>>> https://gist.github.com/garlandkr/4954272 for logstash style output. >>>> >>>> >>>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> [email protected] >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> >> _______________________________________________ >> varnish-misc mailing list >> [email protected] >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
