access logs to parquet

2024-01-11 Thread Mark Lybarger
hi, i'm using nginx as a proxy to api gateway / lambda services. each day, i get 500mb of gzipped access logs from 6 proxy servers. i want to load these nginx access logs into a data lake that takes parquet format as input. my question is fairly general, is there something that easily converts

transforming static files

2020-08-31 Thread Mark Lybarger
i have a bunch of files on a local filesystem (ok, it's NAS) that I serve up using an nginx docker image, just pointing the doc root to the system i want to share. that's fine for my xml files. the users can browse and see then on the filesystem. i also have some .bin files that can be

rewrite ssl proxy retain query string parms

2020-08-13 Thread Mark Lybarger
I'm using rewrite to change some tokens in the url path, and am using ssl proxy to send traffic to a downstream server. if i post to https://myhost/start/foo/213/hello, the request gets to https://client-service-host/client/service/hello/213 using the needed certificate. great. my question is,

lzw compression

2019-08-23 Thread Mark Lybarger
Hi, I have embedded clients using my REST api (HTTP POST/GET etc). We want to be able to compress the client data over the wire so that there are fewer packets. Apparently, in some markets, people still pay by the MB. The embedded client can only support LZW compression due to available