Tyson Norris <tnor...@adobe.com.INVALID> wrote on 08/15/2018 08:29:48 PM: > > FWIW This won’t help with concurrent activations since the logs from > concurrent activations will be interleaved (I think Dave was not > suggesting to use this for concurrent activations). It will only > help in the case where log processing is done outside of the > invoker, and logs are not interleaved from multiple activations.
Agreed. This is the case I'm attempting to optimize for on Kubernetes. Kube clusters typically already have an external logging service deployed and the current KubernetesContainerFactory log processing by the Invoker is a major performance bottleneck. > I’m not sure having a start sentinel is simpler than just including > the activation id in the existing sentinel line (end of log segment, > not the beginning), but it would be probably simpler to read for a human. I think it makes offline processing to add the activationId to every logline more efficient because you will already have the activationId in hand when you read each "real" log line. > If people use blackbox actions, and if blackbox containers have > different log collection than managed actions, I think that would be > a reason to not do anything until there is better support for > structured logging, since if you are still using invoker to collect > blackbox logs, you might as well use it to collect all logs? It may > be that majority log collection is not blackbox so you could get > some efficiencies there, but the added mess of multiple log > collection approaches may bring different problems (my logs behave > different for different types of actions, etc). On large deployments, we segregate blackbox/non-blackbox actions to different invokers. So if all the non-blackbox runtimes are updated to have start sentinels, those invokers can be cut out of the log processing entirely. It's a slightly more complex deployment configuration, but I think it will be worth it (at least on Kubernetes). > > One option might be to allow the /init endpoint to return some > details about the container image, so that it can hint how it > expects logs to be handled (if at all) at the invoker - currently / > init response is only interpreted in case of a non-200 response. > This same approach may be useful for other optional facilities like > support of concurrency or gpu, where the container can signal it’s > support and fail early if there is a mismatch with the action being > executed. This would not resolve the different behavior problem, but > would provide a smooth transition for older blackbox images. > An interesting idea. --dave
