Just like with abstract artifacts (see VirtualFile, ArtifactManager), I think there should be an abstract log. At Yahoo, have some issues with logs being stored on the master. Since the output stream of the log is wrapped by a remote proxy in StreamBuildListener, every log message on the slave gets transferred to the master. That's a lot of throughput. Also, some of our logs get big. That's a lot of storage.
I propose that task/build listeners be made extensible, so that users can have something other than a StreamBuildListener in a build. Further, the log should no longer be file-based; the Listener provides the input and output stream. This should be easy to serialize in the "remote logging service" perspective: just provide the service configuration, and an API call can be made from the slave to publish to the log, and any authorized user of the log can subscribe. The master can either directly subscribe and pipe the log through the UI, or another service which is authorized to subscribe to the logging service can provide the UI externally. One of the tougher things about this change is that it will not be compatible with any BuildWrapper and ConsoleLogFilter plugins. Those plugins decorate the OutputStream on master referred to by the slave's proxy stream. Since the plugins are on the master, it forces the slave to send all log lines to the master for decoration. This pattern tightly couples the logging concept with a master-based file. In order to maintain compatibility for those users for whom this is not an issue, I recommend a "high availability mode" feature flag. This flag will enable certain features that improve availability and scalability, and disable those extension points (BuildWrapper, ConsoleLogFilter) which are incompatible with this new flow. As Jenkins moves towards high availability and scalability, I'm certain there will be more incompatible extension points. I think having that flexibility of choice is good. Those users who aren't hitting that performance ceiling can continue to use Jenkins as-is. Users who are having performance issues can trade-off a bit of sparkle for a more scalable Jenkins. I think that this new logging pattern and feature flag will make that possible. Thank you, -- Larry Bordowitz -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/199146495.3725000.1452708602502.JavaMail.yahoo%40mail.yahoo.com. For more options, visit https://groups.google.com/d/optout.
