Hello Ashish, Thank you for working on this addition. It looks very interesting.
I just have a quick question. From my experience, the userLoginId can contain PII (personally identifiable information) such as the user's name or email address. Would there maybe be a quick way to offer the option to obfuscate or hash those values for the logs? Of course the ability to follow a user's activity through multiple requests is very helpful while debugging and I wouldn't want to lose that option, but it would spare the developers, who have to worry about compliance, one extra headache to have to deal with this matter. I look forward to hearing your thoughts on the matter. Best regards, Konstantinos Marinos On Thu, Jul 9, 2026 at 3:08 PM Ashish Vijaywargiya < [email protected]> wrote: > Hello Lucas(& All), > > Thank you for sharing your thoughts; they were very helpful. 👍 > > Here is the PR of my work: > https://github.com/apache/ofbiz-framework/pull/1443 > > Quick summary of what's done on this branch(ecs-json-logging): > https://github.com/ashishvijaywargiya/ofbiz-framework/tree/ecs-json-logging > > 1. Added an opt-in JSON logging option, on top of the existing log4j logs, > not replacing them. > > 2. It's controlled by a flag in debug.properties - turn on > json.logs.enabled to switch it on. > > 3. Once enabled, it writes a separate ofbiz-json.log file inside > runtime/logs/, alongside the normal log file. > > 4. Each log line now carries requestId/visitId/userLoginId so we can trace > a single request or visit across log lines. > > 5. The JSON format/template is configurable too, so it can be pointed at > different schemas (e.g. Splunk, CloudWatch) later without code changes. > > On my laptop, I've also set up Grafana + Elastic + OpenSearch via Docker, > just watching this ofbiz-json.log file and picking up new entries > automatically > - this is a local setup for trying it out, not part of the PR itself. > > Lucas, yes, this stays strictly opt-in on top of log4j, nothing is > replaced. > > On querying: there's no built-in query interface in the PR itself; querying > is handled externally by tools like Grafana/Elastic/OpenSearch watching the > JSON log file, as I'm testing locally. > > Here are the screenshots of watching the ofbiz logs generated in ECS/JSON > format using the following stacks (Grafana + Elastic + OpenSearch), sharing > it again from the previous email, just for everyone's easy reference: > > Grafana-Loki-Promtail(shared yesterday) > > https://drive.google.com/drive/folders/1PD0BLKa4weAXEf16aegEC2DAfKLcgZhw?usp=sharing > > Elasticsearch-Kibana-Filebeat > > https://drive.google.com/drive/folders/1OvKrQYZZxN4GPlQA9rR6tzU54IGjv_Mf?usp=sharing > > OpenSearch-OpenSearchDashboards-FluentBit > > https://drive.google.com/drive/folders/1cIrnEyPS91u7GR38hdOPwt8JixFhHwdP?usp=drive_link > > Please let me know if you require further details on this work. > > Very soon, I will prepare a document on OFBiz Wiki for this addition and > how anyone can use this feature. > Thank you. > > -- > Kind Regards, > Ashish Vijaywargiya > Vice President of Operations > *HotWax Systems* > *Enterprise open source experts* > http://www.hotwaxsystems.com > > > > On Thu, Jul 9, 2026 at 1:20 PM Lukas Finster <[email protected]> > wrote: > > > Hi Ashish, > > > > it looks interesting and i can see this potentially beeing very usefull. > > > > I definitly would stick to keeping it an opt-in feature on top of the > > existing log4j instead of a replacement. > > > > One question I am wondering about is how querying will work. > > > > Will your PR include some type of interface for querying the logs? > > > > > > Best regards, > > > > Lukas > > > > > > > > Am 09.07.26 um 09:36 schrieb Ashish Vijaywargiya: > > > Hello Dev Members, > > > > > > Quick update on the structured logging work I shared yesterday - the > > > implementation is finalized now and I've tested it end to end. > > > > > > 1. json.logs.enabled and json.logs.template switches added in > > > framework/base/config/debug.properties, default off, default is the > > bundled > > > ECS template. > > > > > > 2. build.gradle reads both and passes them as JVM system properties > > > automatically - plain ./gradlew ofbiz picks them up, no extra flags > > needed. > > > > > > 3. log4j2.xml has one new opt-in JSON appender writing ECS formatted > > lines > > > to runtime/logs/ofbiz-json.log alongside the existing text logs, > nothing > > > else changes. > > > > > > 4. Correlation fields (requestId, visitId, userLoginId) now come from a > > new > > > CorrelationValve in framework/catalina, running as a Tomcat valve ahead > > of > > > the filter chain - ControlFilter stays untouched. > > > > > > 5. Added CorrelationFieldProvider, a ServiceLoader based extension > point > > - > > > any component can add extra fields like security group or permissions > > with > > > zero framework changes. > > > > > > 6. All providers get merged; a provider overwriting a core field is > > allowed > > > but logged as a warning, and a failing provider is caught and skipped > > > without breaking the request. > > > > > > 7. Tested live on a running OFBiz instance - anonymous request, > > > session/visit, and login correlation, plus the provider > > > merge/overwrite/failure cases, all confirmed from real log output. > > > > > > 8. Tested against two separate observability stacks to confirm the > output > > > is genuinely portable - Elastic (Elasticsearch + Kibana + Filebeat) and > > > OpenSearch (OpenSearch + OpenSearch Dashboards + Fluent Bit). > > > > > > 9. Both ingested the same ofbiz-json.log with no changes, and I got a > > > working dashboard (log volume by level plus a live search) with > real-time > > > data in each. > > > > > > 10. Also confirmed the template placeholder is genuinely swappable at > > > runtime through debug.properties, tested with a different schema, not > > just > > > the default ECS one. > > > > > > Here are the screenshots of watching the ofbiz logs generated in > ECS/JSON > > > format using following stacks (Grafana + Elastic + OpenSearch): > > > > > > Grafana-Loki-Promtail(shared yesterday) > > > > > > https://drive.google.com/drive/folders/1PD0BLKa4weAXEf16aegEC2DAfKLcgZhw?usp=sharing > > > > > > Elasticsearch-Kibana-Filebeat > > > > > > https://drive.google.com/drive/folders/1OvKrQYZZxN4GPlQA9rR6tzU54IGjv_Mf?usp=sharing > > > > > > OpenSearch-OpenSearchDashboards-FluentBit > > > > > > https://drive.google.com/drive/folders/1cIrnEyPS91u7GR38hdOPwt8JixFhHwdP?usp=drive_link > > > > > > I will share the PR for this work soon. > > > Thank you. > > > > > > -- > > > Kind Regards, > > > Ashish Vijaywargiya > > > Vice President of Operations > > > *HotWax Systems* > > > *Enterprise open source experts* > > > http://www.hotwaxsystems.com > > > > > > > > > > > > On Wed, Jul 8, 2026 at 9:01 PM Ashish Vijaywargiya < > > > [email protected]> wrote: > > > > > >> Hello OFBiz Dev Members, > > >> > > >> Here I am sharing my notes on which I've been working through: opt-in > > >> structured (ECS - Elastic Common Schema) JSON logging on top of the > > >> existing log4j2.xml. > > >> > > >> 1. Today's OFBiz logs are plain text, so every downstream consumer > needs > > >> its own regex/grok parser before the data is even queryable - that's > the > > >> core problem this addresses. > > >> > > >> 2. Added a JSON appender to the existing > > >> framework/base/config/log4j2.xml, writing ECS (Elastic Common Schema) > > >> formatted lines to runtime/logs/ofbiz-json.log, alongside the current > > >> console/ofbiz.log/error.log output - nothing existing changes shape or > > >> content. > > >> > > >> 3. It's fully opt-in and off by default: a single > > json.logs.enabled=false > > >> switch in framework/base/config/debug.properties. Flip it to true and > > the > > >> default ./gradlew ofbiz command picks it up on its own - no extra flag > > to > > >> remember or document. > > >> > > >> 4. ECS was chosen as the default (and only bundled) schema because > it's > > an > > >> industry standard that Elastic, Grafana, and most log shippers > > (Promtail, > > >> Fluent Bit, Filebeat) already understand out of the box, with no > custom > > >> mapping needed. > > >> > > >> 5. log4j2.xml exposes a jsonLogTemplateUri placeholder (defaulting to > > the > > >> bundled ECS template), overridable via > > >> -Dofbiz.json.logs.template=classpath:your-template.json - so a > > >> Splunk/CloudWatch/anything-else template can live in a separate custom > > >> component and be swapped in with zero framework changes. > > >> > > >> 6. Every log line also carries requestId, visitId, and userLoginId > > >> (populated in ControlFilter via Log4j2's ThreadContext/MDC), so all > > lines > > >> from one HTTP request or one user visit can be pulled with a single > > query - > > >> correlation the current plain-text logs don't support at all. > > >> > > >> 7. JSON-over-file also fits containerized/Kubernetes deployments > better > > >> than relying solely on local rolling text files, which assume > > persistent, > > >> non-ephemeral disk. > > >> > > >> 8. The whole thing lives inside the existing log4j2.xml behind a > > >> Select/SystemPropertyArbiter condition. > > >> > > >> 9. How to test it: Setup a local observability/ stack (Grafana + Loki > + > > >> Promtail via docker-compose) to try this out end-to-end. > > >> > > >> I completed the first pass of the above notes. Here are the > screenshots > > of > > >> watching the ofbiz logs generated in ECS/JSON format using Grafana + > > Loki > > >> + Promtail. > > >> > > >> > > >> > > > https://drive.google.com/file/d/1SJRqFKNT0dtJ5mBP2fsC7YSKY-BP97Wo/view?usp=drive_link > > >> > > >> > > >> > > > https://drive.google.com/file/d/1J6RlrjnQawFFwtu4YO8V2HzlbEJD5hZ-/view?usp=drive_link > > >> > > >> > > >> > > > https://drive.google.com/file/d/1AKsdcGg-N1WL7k2uClUdilQoOu2P3MLL/view?usp=drive_link > > >> > > >> > > >> > > > https://drive.google.com/file/d/181f5kSNwfCieA_z79Z2IEjw0O7PgIPWX/view?usp=drive_link > > >> > > >> Here are the reference documents that I considered: > > >> > > >> - Elastic Common Schema (ECS) spec: > > >> https://www.elastic.co/guide/en/ecs/current/index.html > > >> > > >> - ECS field reference: > > >> https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html > > >> > > >> - ECS GitHub repo: https://github.com/elastic/ecs > > >> > > >> - Log4j2 JsonTemplateLayout (the mechanism used here, includes the > > bundled > > >> ECS template): > > >> https://logging.apache.org/log4j/2.x/manual/json-template-layout.html > > >> > > >> I will be sharing a PR soon for everyone's reference. > > >> > > >> Please let me know your thoughts on this feature. > > >> Thank you. > > >> > > >> -- > > >> Kind Regards, > > >> Ashish Vijaywargiya > > >> Vice President of Operations > > >> *HotWax Systems* > > >> *Enterprise open source experts* > > >> http://www.hotwaxsystems.com > > >> > > >> > > -- > > Lukas Finster > > Softwareentwickler & Berater > > > > ecomify GmbH, Stralsunder Straße 63, 33605 Bielefeld > > Fon: +49 521 448157-90 | Fax: +49 521 448157-99 | www.ecomify.de > > Court Registration: Amtsgericht Bielefeld, HRB 41683 | CEO: Martin > Becker, > > Michael Brohl > > > > >
