Re: [I] [Feature] Implement Inverted Index for SeriesDatabase [skywalking]
lujiajing1126 commented on issue #11496: URL: https://github.com/apache/skywalking/issues/11496#issuecomment-1793326932 > please assign to me, thanks. Assigned to u -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support printing TID to logs [skywalking-python]
wu-sheng merged PR #323: URL: https://github.com/apache/skywalking-python/pull/323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking-python) branch master updated: Add support printing TID to logs (#323)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking-python.git
The following commit(s) were added to refs/heads/master by this push:
new bea712c Add support printing TID to logs (#323)
bea712c is described below
commit bea712c7a87583432764ec5e176bca49b9595bb3
Author: Starry
AuthorDate: Sat Nov 4 11:29:01 2023 +0800
Add support printing TID to logs (#323)
---
CHANGELOG.md | 1 +
docs/en/setup/advanced/LogReporter.md | 8 +++-
skywalking/agent/__init__.py | 2 --
skywalking/log/sw_logging.py | 3 +++
4 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 849e94e..c775273 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
- Feature:
- Users now can specify the `SW_AGENT_ASYNCIO_ENHANCEMENT` environment
variable to enable the performance enhancement with asyncio (#316)
+ - Support printing Trace IDs (TID) to collected application logs (#323)
- Plugins:
- Add neo4j plugin.(#312)
diff --git a/docs/en/setup/advanced/LogReporter.md
b/docs/en/setup/advanced/LogReporter.md
index c1982d9..e08e65f 100644
--- a/docs/en/setup/advanced/LogReporter.md
+++ b/docs/en/setup/advanced/LogReporter.md
@@ -103,4 +103,10 @@ Will result in:
}
]
}
-```
\ No newline at end of file
+```
+
+## Print trace ID in your logs
+To print out the trace IDs in the logs, simply add `%(tid)s` to the
`agent_log_reporter_layout`.
+
+You can take advantage of this feature to print out the trace IDs on any
channel you desire, not limited to reporting logs to OAP,
+this can be achieved by using any formatter you prefer in your own application
logic.
\ No newline at end of file
diff --git a/skywalking/agent/__init__.py b/skywalking/agent/__init__.py
index fc7f9e7..9ee4d92 100644
--- a/skywalking/agent/__init__.py
+++ b/skywalking/agent/__init__.py
@@ -268,7 +268,6 @@ class SkyWalkingAgent(Singleton):
logger.info(f'SkyWalking sync agent instance
{config.agent_instance_name} starting in pid-{os.getpid()}.')
# Install log reporter core
-# TODO - Add support for printing traceID/ context in logs
if config.agent_log_reporter_active:
from skywalking import log
log.install()
@@ -502,7 +501,6 @@ class SkyWalkingAgentAsync(Singleton):
self._finished = asyncio.Event()
# Install log reporter core
-# TODO - Add support for printing traceID/ context in logs
if config.agent_log_reporter_active:
from skywalking import log
log.install()
diff --git a/skywalking/log/sw_logging.py b/skywalking/log/sw_logging.py
index 801befe..e192368 100644
--- a/skywalking/log/sw_logging.py
+++ b/skywalking/log/sw_logging.py
@@ -75,6 +75,9 @@ def install():
context = get_context()
+if '%(tid)s' in layout:
+record.tid = str(context.segment.related_traces[0])
+
active_span_id = -1
primary_endpoint_name = ''
Re: [I] [Feature] Implement Inverted Index for SeriesDatabase [skywalking]
zesiar0 commented on issue #11496: URL: https://github.com/apache/skywalking/issues/11496#issuecomment-1793321694 please assign to me, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking-python) branch master updated: loose restrict of greenlet (#3) (#326)
This is an automated email from the ASF dual-hosted git repository.
yihaochen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking-python.git
The following commit(s) were added to refs/heads/master by this push:
new 9b1acaa loose restrict of greenlet (#3) (#326)
9b1acaa is described below
commit 9b1acaa1fb9adb56a01df96b41628a51c66455f9
Author: Jay Zhou <4192339+jaych...@users.noreply.github.com>
AuthorDate: Sat Nov 4 10:58:43 2023 +0800
loose restrict of greenlet (#3) (#326)
---
poetry.lock | 235 --
pyproject.toml| 4 +-
skywalking/profile/profile_context.py | 15 +-
tests/e2e/base/Dockerfile.e2e | 2 +-
tests/e2e/case/profiling/greenlet/start_gevent.sh | 4 +-
tests/plugin/web/sw_flask/docker-compose.yml | 4 +-
6 files changed, 140 insertions(+), 124 deletions(-)
diff --git a/poetry.lock b/poetry.lock
index 917ae03..52f500a 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.5.1 and should not be
changed by hand.
+# This file is automatically @generated by Poetry 1.6.1 and should not be
changed by hand.
[[package]]
name = "aiofiles"
@@ -1071,51 +1071,68 @@ tomli = ">=1.1.0"
[[package]]
name = "gevent"
-version = "22.8.0"
+version = "22.10.2"
description = "Coroutine-based network library"
optional = false
python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5"
files = [
-{file = "gevent-22.8.0-cp27-cp27m-macosx_10_14_x86_64.whl", hash =
"sha256:117d947d3a6d93e27ba0ce3c8de6c49e33e106ce8841ff34777bdb2c884605f7"},
-{file = "gevent-22.8.0-cp27-cp27m-win32.whl", hash =
"sha256:94174ff3e4d16f37a295437cc17d1ce22e5bd30059eecaff10a28bb5724da1e3"},
-{file = "gevent-22.8.0-cp27-cp27m-win_amd64.whl", hash =
"sha256:c78d64845186b7d2f940a27c466bd5a36c20acf9972eb9b3e3566052942c4af3"},
-{file = "gevent-22.8.0-cp310-cp310-macosx_10_15_x86_64.whl", hash =
"sha256:f0b7017c58a7d61649d74049b46851e5024c95b73c18144092be0e348a198e39"},
-{file =
"gevent-22.8.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl",
hash =
"sha256:5d71144d79c08dc1e0554e8b142500d72e40feedc596657b25d87940bedca0d7"},
-{file =
"gevent-22.8.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:9e0d6fa493d9e48d1d36c2bbca4231852e8aa70d4c22d76c11a438a61cdd730a"},
-{file = "gevent-22.8.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash =
"sha256:646777ee668f745cad7671fa20ca5384b2baa2709562399caf433f4a52967081"},
-{file = "gevent-22.8.0-cp310-cp310-win_amd64.whl", hash =
"sha256:27ea8edae95b530e86476df59c785d96a2eb45d0dd85335dccd474045278b1d6"},
-{file = "gevent-22.8.0-cp311-cp311-macosx_10_9_universal2.whl", hash =
"sha256:594565ab5b7ad95e76ef66d53b434ca22848b5154da817d6f4e43603b548f71c"},
-{file =
"gevent-22.8.0-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash
= "sha256:e38f76f00eb489b2e87fd67896e48a3127b2d613c78a4930b25820c42e884ef4"},
-{file =
"gevent-22.8.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:e2e82f09ab81551623650a1b0315ed790f82162d6ef2e0ed9c7cce50398c2b09"},
-{file = "gevent-22.8.0-cp36-cp36m-musllinux_1_1_x86_64.whl", hash =
"sha256:df50d390785db12a0825796654a7d8ae2263c46419de72daaac92a1aa21895c8"},
-{file = "gevent-22.8.0-cp36-cp36m-win32.whl", hash =
"sha256:aced6fe003aceb0d816ec27eced385adbf838f190b73136567eaaad2c3777c8a"},
-{file = "gevent-22.8.0-cp36-cp36m-win_amd64.whl", hash =
"sha256:6301302095f5e30237eb135198e97a2cdbb822edaac05fcf9d4223b19ee65dd5"},
-{file = "gevent-22.8.0-cp37-cp37m-macosx_10_15_x86_64.whl", hash =
"sha256:79050166a526f73352f8bbbdc246ae7fd233d24e4eea9be8793cc67021f1a967"},
-{file =
"gevent-22.8.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash
= "sha256:9c909b54f803585e602102ec71f23c6ad2ee69fca3ee0bfdb99d694ff76527f5"},
-{file =
"gevent-22.8.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:f895f56c9cf132fa4f62950ef960b68d852762344f16475724bdd978e4d856eb"},
-{file = "gevent-22.8.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash =
"sha256:c606f7839d45dafd90ae87359e1f7f2b795b2e4fcecd4632f1edd0cd01f6b439"},
-{file = "gevent-22.8.0-cp37-cp37m-win32.whl", hash =
"sha256:c92129d3e29608e95796863f7c88dfe61a341386a8a882c70ff5684370fd2bd2"},
-{file = "gevent-22.8.0-cp37-cp37m-win_amd64.whl", hash =
"sha256:bd2536a45d3204fa2b13885ef4342bb5e3f0199916be5b4c728d53da9da57364"},
-{file = "gevent-22.8.0-cp38-cp38-macosx_10_15_x86_64.whl", hash =
"sha256:7f2256efb270f02855006254d3a10774f57c5b78c59e79fce9bbd6027a800598"},
-{file =
"gevent-22.8.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash
= "sha256:004e1b58dd2d1f1622ec566864bcd9a807461d28115916ac76eb54fde42460a1"},
-{file =
Re: [PR] loose restrict of greenlet (#3) [skywalking-python]
Superskyyy merged PR #326: URL: https://github.com/apache/skywalking-python/pull/326 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] loose restrict of greenlet (#3) [skywalking-python]
Superskyyy commented on PR #326: URL: https://github.com/apache/skywalking-python/pull/326#issuecomment-1793315758 Thanks for the PR! @jaychoww -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] loose restrict of greenlet (#3) [skywalking-python]
Superskyyy commented on PR #326: URL: https://github.com/apache/skywalking-python/pull/326#issuecomment-1793315683 Ok, let's merge this first, few related issues seem to be reported to psutil too, I will continue to monitor that and loosen the version requirements once fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix: some links are broken in the doc [skywalking-banyandb]
hanahmily commented on PR #350: URL: https://github.com/apache/skywalking-banyandb/pull/350#issuecomment-1792527045 It's nice to see the dead link checker work as expected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature] Optimize Index Definition for BanyanDB Module in OAP [skywalking]
wu-sheng commented on issue #11497: URL: https://github.com/apache/skywalking/issues/11497#issuecomment-1792514850 So, this is only OAP side code change, and the protocol between OAP and DB is stable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
weixiang1862 commented on PR #652: URL: https://github.com/apache/skywalking-website/pull/652#issuecomment-1792390377 > @weixiang1862 If you are going to make Nginx monitoring available on the upstream, you should follow these > > 1. Consider how Nginx metrics(Prometheus -> OTEL maybe?) to SkyWalking to indicate metrics. > 2. Create menu and layer for Nginx > 3. Ship logs in to Nginx layer service, and keep the name consistent with the metrics. > 4. Take APISIX monitoring as a good reference. > > And, please create an issue to track this when you begin to work on. Ok, once I begin I will create an issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix: some links are broken in the doc [skywalking-banyandb]
wu-sheng commented on PR #350: URL: https://github.com/apache/skywalking-banyandb/pull/350#issuecomment-1792387081 > There is an image URL is broken. Is there anyone know it? > > FILE: ./docs/concept/persistence-storage.md [✖] https://skywalking.apache.org/doc-graph/banyandb/v0.5.0/io_uring.png It is a jpg. Ref URL, https://skywalking.apache.org/doc-graph/banyandb/v0.5.0/io_uring.jpg The original file is hosted here, https://github.com/apache/skywalking-website/blob/master/static/doc-graph/banyandb/v0.5.0/io_uring.jpg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
wu-sheng commented on PR #652: URL: https://github.com/apache/skywalking-website/pull/652#issuecomment-1792383240 @weixiang1862 If you are going to make Nginx monitoring available on the upstream, you should follow these 1. Consider how Nginx metrics(Prometheus -> OTEL maybe?) to SkyWalking to indicate metrics. 2. Create menu and layer for Nginx 3. Ship logs in to Nginx layer service, and keep the name consistent with the metrics. 4. Take APISIX monitoring as a good reference. And, please create an issue to track this when you begin to work on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking-website) branch asf-site updated: deploy: 0180c702e7468694eee3979252ae4abcbdbdba24
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/skywalking-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 9112397865e deploy: 0180c702e7468694eee3979252ae4abcbdbdba24
9112397865e is described below
commit 9112397865eb0663eb75a7cbaf823ac1377231ce
Author: wu-sheng
AuthorDate: Fri Nov 3 10:06:00 2023 +
deploy: 0180c702e7468694eee3979252ae4abcbdbdba24
---
index.json | 2 +-
team/index.html | 20 ++--
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/index.json b/index.json
index 3a949472c44..39b46649be1 100644
--- a/index.json
+++ b/index.json
@@ -1 +1 @@
-[{"body":"","excerpt":"","ref":"/tags/lal/","title":"LAL"},{"body":"","excerpt":"","ref":"/tags/logging/","title":"Logging"},{"body":"","excerpt":"","ref":"/tags/nginx/","title":"Nginx"},{"body":"","excerpt":"","ref":"/tags/","title":"Tags"},{"body":"背景介绍
Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access
log，我们可以实现对 Nginx 中接口状态的监控。\n在本案例中，将由 fluent-bit 收集 access log，并通过 HTTP
将日志信息发送给 SkyWalking OAP Server 进行进一步的分析。\n环境准备 实验需要的 Nginx 及 Fluent-bit
相关配置文件都被上传到了Github，有需要的读 [...]
\ No newline at end of file
+[{"body":"","excerpt":"","ref":"/tags/lal/","title":"LAL"},{"body":"","excerpt":"","ref":"/tags/logging/","title":"Logging"},{"body":"","excerpt":"","ref":"/tags/nginx/","title":"Nginx"},{"body":"","excerpt":"","ref":"/tags/","title":"Tags"},{"body":"背景介绍
Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access
log，我们可以实现对 Nginx 中接口状态的监控。\n在本案例中，将由 fluent-bit 收集 access log，并通过 HTTP
将日志信息发送给 SkyWalking OAP Server 进行进一步的分析。\n环境准备 实验需要的 Nginx 及 Fluent-bit
相关配置文件都被上传到了Github，有需要的读 [...]
\ No newline at end of file
diff --git a/team/index.html b/team/index.html
index aa2d4536c98..7f3d1bfe124 100644
--- a/team/index.html
+++ b/team/index.html
@@ -1066,9 +1066,9 @@ The SkyWalking team is comprised of Members and
Contributors, and the growth has
-https://github.com/yswdqz; target="_blank">
+https://github.com/weixiang1862;
target="_blank">
- yswdqz
+ weixiang1862
@@ -1076,9 +1076,9 @@ The SkyWalking team is comprised of Members and
Contributors, and the growth has
-https://github.com/Humbertzhang;
target="_blank">
+https://github.com/yswdqz; target="_blank">
- Humbertzhang
+ yswdqz
@@ -1086,9 +1086,9 @@ The SkyWalking team is comprised of Members and
Contributors, and the growth has
-https://github.com/FingerLeader;
target="_blank">
+https://github.com/Humbertzhang;
target="_blank">
- FingerLeader
+ Humbertzhang
@@ -1096,9 +1096,9 @@ The SkyWalking team is comprised of Members and
Contributors, and the growth has
-https://github.com/gxthrj; target="_blank">
+https://github.com/FingerLeader;
target="_blank">
- gxthrj
+ FingerLeader
@@ -1106,9 +1106,9 @@ The SkyWalking team is comprised of Members and
Contributors, and the growth has
-https://github.com/weixiang1862;
target="_blank">
+https://github.com/gxthrj; target="_blank">
- weixiang1862
+ gxthrj
Re: [PR] fix: some links are broken in the doc [skywalking-banyandb]
LinuxSuRen commented on PR #350: URL: https://github.com/apache/skywalking-banyandb/pull/350#issuecomment-1792124964 There is an image URL is broken. Is there anyone know it? FILE: ./docs/concept/persistence-storage.md [✖] https://skywalking.apache.org/doc-graph/banyandb/v0.5.0/io_uring.png -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] PHP agent don't have any endpoint info [skywalking]
aclstack commented on issue #11488: URL: https://github.com/apache/skywalking/issues/11488#issuecomment-1792089894 but if use docker image skyapm/skywalking-php:v4.2.0-7.2-fpm-alpine the service endpoints can display  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
weixiang1862 commented on PR #652: URL: https://github.com/apache/skywalking-website/pull/652#issuecomment-1792062269 Thanks all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix: some links are broken in the doc [skywalking-banyandb]
codecov-commenter commented on PR #350: URL: https://github.com/apache/skywalking-banyandb/pull/350#issuecomment-1792054279 ## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/350?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging [#350](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/350?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) (b397c77) into [main](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/222ca556c2a1aa974145ad6f2fefcf612bd68174?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) (222ca55) will **decrease** coverage by `0.02%`. > The diff coverage is `n/a`. ```diff @@Coverage Diff @@ ## main #350 +/- ## == - Coverage 38.25% 38.23% -0.02% == Files 117 117 Lines 1283112831 == - Hits 4908 4906 -2 - Misses 7442 7444 +2 Partials 481 481 ``` [see 1 file with indirect coverage changes](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/350/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) :mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/codecov/) today! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix: some links are broken in the doc [skywalking-banyandb]
LinuxSuRen commented on PR #350: URL: https://github.com/apache/skywalking-banyandb/pull/350#issuecomment-1792051978 > Could you refer main repo to add dead link check? Sure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix: some links are broken in the doc [skywalking-banyandb]
wu-sheng commented on PR #350: URL: https://github.com/apache/skywalking-banyandb/pull/350#issuecomment-1792047583 Could you refer main repo to add dead link check? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
rootsongjc commented on PR #652: URL: https://github.com/apache/skywalking-website/pull/652#issuecomment-1792031220 @wu-sheng I will post this blog to the Cloud Native Community WeChat official account as well as the [SkyWalking Go Toolkit Trace 详解](https://skywalking.apache.org/zh/2023-10-18-skywalking-toolkit-trace/) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] fix: some links are broken in the doc [skywalking-banyandb]
LinuxSuRen opened a new pull request, #350: URL: https://github.com/apache/skywalking-banyandb/pull/350 - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Fixes apache/skywalking#. - [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking-banyandb/blob/main/CHANGES.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
wu-sheng commented on PR #652: URL: https://github.com/apache/skywalking-website/pull/652#issuecomment-1792020363 @kezhenxu94 Could you copost this on juejin? @rootsongjc Consider this too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking-website) branch master updated: Add blog: collect and analyse nginx access log by LAL. (#652)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking-website.git
The following commit(s) were added to refs/heads/master by this push:
new 0180c702e74 Add blog: collect and analyse nginx access log by LAL.
(#652)
0180c702e74 is described below
commit 0180c702e7468694eee3979252ae4abcbdbdba24
Author: weixiang1862 <652048...@qq.com>
AuthorDate: Fri Nov 3 16:05:19 2023 +0800
Add blog: collect and analyse nginx access log by LAL. (#652)
---
.../index.md | 157 +
.../nginx-log.jpg | Bin 0 -> 152586 bytes
.../nginx-metric.jpg | Bin 0 -> 93189 bytes
3 files changed, 157 insertions(+)
diff --git
a/content/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/index.md
b/content/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/index.md
new file mode 100644
index 000..0fc6900eb7a
--- /dev/null
+++ b/content/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/index.md
@@ -0,0 +1,157 @@
+---
+title: "使用 LAL 收集并分析 Nginx access log"
+author: "魏翔"
+date: 2023-10-29
+description: "本篇文章演示如何将 Nginx access log 收集到 SkyWalking 中，并通过 LAL 进行指标分析"
+tags:
+- LAL
+- Logging
+- Nginx
+---
+
+## 背景介绍
+Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access
log，我们可以实现对 Nginx 中接口状态的监控。
+
+在本案例中，将由 [fluent-bit](https://fluentbit.io/) 收集 access log，并通过 HTTP 将日志信息发送给
SkyWalking OAP Server 进行进一步的分析。
+
+## 环境准备
+实验需要的 Nginx 及 Fluent-bit
相关配置文件都被上传到了[Github](https://github.com/weixiang1862/nginx-fluent-bit)，有需要的读者可以自行
git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。
+
+### Nginx日志格式配置
+LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.
+
+```
+http {
+...
+...
+
+log_format main '{"remote_addr": "$remote_addr",'
+'"remote_user": "$remote_user",'
+'"request": "$request",'
+'"time": "$time_iso8601",'
+'"status": "$status",'
+'"request_time":"$request_time",'
+'"body_bytes_sent": "$body_bytes_sent",'
+'"http_referer": "$http_referer",'
+'"http_user_agent": "$http_user_agent",'
+'"http_x_forwarded_for": "$http_x_forwarded_for"}';
+
+access_log /var/log/nginx/access.log main;
+
+...
+...
+}
+```
+
+### Fluent bit Filter
+我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：
+- body：日志内容体
+- service：服务名称
+- serviceInstance：实例名称
+```
+function rewrite_body(tag, timestamp, record)
+local newRecord = {}
+newRecord["body"] = { json = { json = record.log } }
+newRecord["service"] = "nginx::nginx"
+newRecord["serviceInstance"] = "localhost"
+return 1, timestamp, newRecord
+end
+```
+
+## OAP 日志分析
+### LAL定义
+在 filter 中，我们通过条件判断，只处理 `service=nginx::nginx` 的服务，其他服务依旧走默认逻辑：
+
+第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。
+
+第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log
json 中的 time。
+
+第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。
+
+第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：
+- `nginx_log_count`：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。
+- `nginx_request_time`：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。
+- `nginx_body_bytes_sent`：body 大小指标可以帮助我们了解网关上的流量情况。
+- `nginx_status_code`：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。
+
+```yaml
+rules:
+ - name: default
+layer: GENERAL
+dsl: |
+ filter {
+if (log.service == "nginx::nginx") {
+ json {
+abortOnFailure true
+ }
+
+ extractor {
+timestamp parsed.time as String, "-MM-dd'T'HH:mm:ssXXX"
+tag status: parsed.status
+tag remote_addr: parsed.remote_addr
+
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_log_count"
+ value 1
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_request_time"
+ value parsed.request_time as Double
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_body_bytes_sent"
+ value parsed.body_bytes_sent as Long
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance,
status: parsed.status
+ name "nginx_status_code"
+ value 1
+}
+ }
+}
+
+sink {
+}
+ }
+```
+经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
wu-sheng merged PR #652: URL: https://github.com/apache/skywalking-website/pull/652 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
weixiang1862 commented on code in PR #652:
URL:
https://github.com/apache/skywalking-website/pull/652#discussion_r1381241981
##
content/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/index.md:
##
@@ -0,0 +1,157 @@
+---
+title: "使用 LAL 收集并分析 Nginx access log"
+author: "魏翔"
+date: 2023-10-29
+description: "本篇文章演示如何将 Nginx access log 收集到 SkyWalking 中，并通过 LAL 进行指标分析"
+tags:
+- LAL
+- Logging
+- Nginx
+---
+
+## 背景介绍
+Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access
log，我们可以实现对 Nginx 中接口状态的监控。
+
+在本案例中，将由 [fluent-bit](https://fluentbit.io/) 收集 access log，并通过 HTTP 将日志信息发送给
SkyWalking OAP Server 进行进一步的分析。
+
+## 环境准备
+实验需要的 Nginx 及 Fluent-bit
相关配置文件都被上传到了[Github](https://github.com/weixiang1862/nginx-fluent-bit)，有需要的读者可以自行
git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。
+
+### Nginx日志格式配置
+LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.
+
+```
+http {
+...
+...
+
+log_format main '{"remote_addr": "$remote_addr",'
+'"remote_user": "$remote_user",'
+'"request": "$request",'
+'"time": "$time_iso8601",'
+'"status": "$status",'
+'"request_time":"$request_time",'
+'"body_bytes_sent": "$body_bytes_sent",'
+'"http_referer": "$http_referer",'
+'"http_user_agent": "$http_user_agent",'
+'"http_x_forwarded_for": "$http_x_forwarded_for"}';
+
+access_log /var/log/nginx/access.log main;
+
+...
+...
+}
+```
+
+### Fluent bit Filter
+我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：
+- body：日志内容体
+- service：服务名称
+- serviceInstance：实例名称
+```
+function rewrite_body(tag, timestamp, record)
+local newRecord = {}
+newRecord["body"] = { json = { json = record.log } }
+newRecord["service"] = "nginx::nginx"
+newRecord["serviceInstance"] = "localhost"
+return 1, timestamp, newRecord
+end
+```
+
+## OAP 日志分析
+### LAL定义
+在 filter 中，我们通过条件判断，只处理 `service=nginx::nginx` 的服务，其他服务依旧走默认逻辑：
+
+第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。
+
+第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log
json 中的 time。
+
+第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。
+
+第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：
+- `nginx_log_count`：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。
+- `nginx_request_time`：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。
+- `nginx_body_bytes_sent`：body 大小指标可以帮助我们了解网关上的流量情况。
+- `nginx_status_code`：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。
+
+```yaml
+rules:
+ - name: default
+layer: GENERAL
+dsl: |
+ filter {
+if (log.service == "nginx::nginx") {
+ json {
+abortOnFailure true
+ }
+
+ extractor {
+timestamp parsed.time as String, "-MM-dd'T'HH:mm:ssXXX"
+tag status: parsed.status
+tag remote_addr: parsed.remote_addr
+
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_log_count"
+ value 1
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_request_time"
+ value parsed.request_time as Double
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_body_bytes_sent"
+ value parsed.body_bytes_sent as Long
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance,
status: parsed.status
+ name "nginx_status_code"
+ value 1
+}
+ }
+}
+
+sink {
+}
+ }
+```
+经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL 中提取的指标进行进一步分析：
+
+
+
+### MAL定义
+在 MAL 中，我们可以对上一步 LAL 中提取的指标进行进一步的分析聚合，下面的例子里：
+
+nginx_log_count、nginx_request_time、nginx_status_code 使用 sum 聚合函数处理，并使用 SUM 方式
downsampling，
+
+nginx_request_time 使用 avg 聚合函数求平均值，默认使用 AVG 方式 downsampling。
+
+完成聚合分析后，SkyWalking Meter System 会完成对上述指标的持久化。
+
+```yaml
+expSuffix: service(['service'], Layer.GENERAL)
+metricPrefix: nginx
+metricsRules:
+ - name: cpm
+exp: nginx_log_count.sum(['service']).downsampling(SUM)
+ - name: avg_request_time
+exp: nginx_request_time.avg(['service'])
+ - name: body_bytes_sent_count
+exp: nginx_body_bytes_sent.sum(['service']).downsampling(SUM)
+ - name: status_code_count
+exp: nginx_status_code.sum(['service','status']).downsampling(SUM)
+```
+最后，我们便可以来到 SkyWalking UI 页面新建 Nginx 仪表板，使用刚刚 MAL 中定义的指标信息创建 Nginx Dashboard：
Review Comment:
> Or a screenshot?
>
> We planned to provide official Nginx
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
weixiang1862 commented on code in PR #652:
URL:
https://github.com/apache/skywalking-website/pull/652#discussion_r1381241585
##
content/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/index.md:
##
@@ -0,0 +1,157 @@
+---
+title: "使用 LAL 收集并分析 Nginx access log"
+author: "魏翔"
+date: 2023-10-29
+description: "本篇文章演示如何将 Nginx access log 收集到 SkyWalking 中，并通过 LAL 进行指标分析"
+tags:
+- LAL
+- Logging
+- Nginx
+---
+
+## 背景介绍
+Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access
log，我们可以实现对 Nginx 中接口状态的监控。
+
+在本案例中，将由 [fluent-bit](https://fluentbit.io/) 收集 access log，并通过 HTTP 将日志信息发送给
SkyWalking OAP Server 进行进一步的分析。
+
+## 环境准备
+实验需要的 Nginx 及 Fluent-bit
相关配置文件都被上传到了[Github](https://github.com/weixiang1862/nginx-fluent-bit)，有需要的读者可以自行
git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。
+
+### Nginx日志格式配置
+LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.
+
+```
+http {
+...
+...
+
+log_format main '{"remote_addr": "$remote_addr",'
+'"remote_user": "$remote_user",'
+'"request": "$request",'
+'"time": "$time_iso8601",'
+'"status": "$status",'
+'"request_time":"$request_time",'
+'"body_bytes_sent": "$body_bytes_sent",'
+'"http_referer": "$http_referer",'
+'"http_user_agent": "$http_user_agent",'
+'"http_x_forwarded_for": "$http_x_forwarded_for"}';
+
+access_log /var/log/nginx/access.log main;
+
+...
+...
+}
+```
+
+### Fluent bit Filter
+我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：
+- body：日志内容体
+- service：服务名称
+- serviceInstance：实例名称
+```
+function rewrite_body(tag, timestamp, record)
+local newRecord = {}
+newRecord["body"] = { json = { json = record.log } }
+newRecord["service"] = "nginx::nginx"
+newRecord["serviceInstance"] = "localhost"
+return 1, timestamp, newRecord
+end
+```
+
+## OAP 日志分析
+### LAL定义
+在 filter 中，我们通过条件判断，只处理 `service=nginx::nginx` 的服务，其他服务依旧走默认逻辑：
+
+第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。
+
+第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log
json 中的 time。
+
+第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。
+
+第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：
+- `nginx_log_count`：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。
+- `nginx_request_time`：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。
+- `nginx_body_bytes_sent`：body 大小指标可以帮助我们了解网关上的流量情况。
+- `nginx_status_code`：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。
+
+```yaml
+rules:
+ - name: default
+layer: GENERAL
+dsl: |
+ filter {
+if (log.service == "nginx::nginx") {
+ json {
+abortOnFailure true
+ }
+
+ extractor {
+timestamp parsed.time as String, "-MM-dd'T'HH:mm:ssXXX"
+tag status: parsed.status
+tag remote_addr: parsed.remote_addr
+
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_log_count"
+ value 1
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_request_time"
+ value parsed.request_time as Double
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_body_bytes_sent"
+ value parsed.body_bytes_sent as Long
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance,
status: parsed.status
+ name "nginx_status_code"
+ value 1
+}
+ }
+}
+
+sink {
+}
+ }
+```
+经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL 中提取的指标进行进一步分析：
+
+
+
+### MAL定义
+在 MAL 中，我们可以对上一步 LAL 中提取的指标进行进一步的分析聚合，下面的例子里：
+
+nginx_log_count、nginx_request_time、nginx_status_code 使用 sum 聚合函数处理，并使用 SUM 方式
downsampling，
+
+nginx_request_time 使用 avg 聚合函数求平均值，默认使用 AVG 方式 downsampling。
+
+完成聚合分析后，SkyWalking Meter System 会完成对上述指标的持久化。
+
+```yaml
+expSuffix: service(['service'], Layer.GENERAL)
+metricPrefix: nginx
+metricsRules:
+ - name: cpm
+exp: nginx_log_count.sum(['service']).downsampling(SUM)
+ - name: avg_request_time
+exp: nginx_request_time.avg(['service'])
+ - name: body_bytes_sent_count
+exp: nginx_body_bytes_sent.sum(['service']).downsampling(SUM)
+ - name: status_code_count
+exp: nginx_status_code.sum(['service','status']).downsampling(SUM)
+```
+最后，我们便可以来到 SkyWalking UI 页面新建 Nginx 仪表板，使用刚刚 MAL 中定义的指标信息创建 Nginx Dashboard：
Review Comment:
> It'd be very helpful if you can also share the sample dashboard
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
wu-sheng commented on code in PR #652:
URL:
https://github.com/apache/skywalking-website/pull/652#discussion_r1381233349
##
content/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/index.md:
##
@@ -0,0 +1,157 @@
+---
+title: "使用 LAL 收集并分析 Nginx access log"
+author: "魏翔"
+date: 2023-10-29
+description: "本篇文章演示如何将 Nginx access log 收集到 SkyWalking 中，并通过 LAL 进行指标分析"
+tags:
+- LAL
+- Logging
+- Nginx
+---
+
+## 背景介绍
+Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access
log，我们可以实现对 Nginx 中接口状态的监控。
+
+在本案例中，将由 [fluent-bit](https://fluentbit.io/) 收集 access log，并通过 HTTP 将日志信息发送给
SkyWalking OAP Server 进行进一步的分析。
+
+## 环境准备
+实验需要的 Nginx 及 Fluent-bit
相关配置文件都被上传到了[Github](https://github.com/weixiang1862/nginx-fluent-bit)，有需要的读者可以自行
git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。
+
+### Nginx日志格式配置
+LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.
+
+```
+http {
+...
+...
+
+log_format main '{"remote_addr": "$remote_addr",'
+'"remote_user": "$remote_user",'
+'"request": "$request",'
+'"time": "$time_iso8601",'
+'"status": "$status",'
+'"request_time":"$request_time",'
+'"body_bytes_sent": "$body_bytes_sent",'
+'"http_referer": "$http_referer",'
+'"http_user_agent": "$http_user_agent",'
+'"http_x_forwarded_for": "$http_x_forwarded_for"}';
+
+access_log /var/log/nginx/access.log main;
+
+...
+...
+}
+```
+
+### Fluent bit Filter
+我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：
+- body：日志内容体
+- service：服务名称
+- serviceInstance：实例名称
+```
+function rewrite_body(tag, timestamp, record)
+local newRecord = {}
+newRecord["body"] = { json = { json = record.log } }
+newRecord["service"] = "nginx::nginx"
+newRecord["serviceInstance"] = "localhost"
+return 1, timestamp, newRecord
+end
+```
+
+## OAP 日志分析
+### LAL定义
+在 filter 中，我们通过条件判断，只处理 `service=nginx::nginx` 的服务，其他服务依旧走默认逻辑：
+
+第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。
+
+第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log
json 中的 time。
+
+第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。
+
+第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：
+- `nginx_log_count`：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。
+- `nginx_request_time`：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。
+- `nginx_body_bytes_sent`：body 大小指标可以帮助我们了解网关上的流量情况。
+- `nginx_status_code`：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。
+
+```yaml
+rules:
+ - name: default
+layer: GENERAL
+dsl: |
+ filter {
+if (log.service == "nginx::nginx") {
+ json {
+abortOnFailure true
+ }
+
+ extractor {
+timestamp parsed.time as String, "-MM-dd'T'HH:mm:ssXXX"
+tag status: parsed.status
+tag remote_addr: parsed.remote_addr
+
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_log_count"
+ value 1
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_request_time"
+ value parsed.request_time as Double
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_body_bytes_sent"
+ value parsed.body_bytes_sent as Long
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance,
status: parsed.status
+ name "nginx_status_code"
+ value 1
+}
+ }
+}
+
+sink {
+}
+ }
+```
+经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL 中提取的指标进行进一步分析：
+
+
+
+### MAL定义
+在 MAL 中，我们可以对上一步 LAL 中提取的指标进行进一步的分析聚合，下面的例子里：
+
+nginx_log_count、nginx_request_time、nginx_status_code 使用 sum 聚合函数处理，并使用 SUM 方式
downsampling，
+
+nginx_request_time 使用 avg 聚合函数求平均值，默认使用 AVG 方式 downsampling。
+
+完成聚合分析后，SkyWalking Meter System 会完成对上述指标的持久化。
+
+```yaml
+expSuffix: service(['service'], Layer.GENERAL)
+metricPrefix: nginx
+metricsRules:
+ - name: cpm
+exp: nginx_log_count.sum(['service']).downsampling(SUM)
+ - name: avg_request_time
+exp: nginx_request_time.avg(['service'])
+ - name: body_bytes_sent_count
+exp: nginx_body_bytes_sent.sum(['service']).downsampling(SUM)
+ - name: status_code_count
+exp: nginx_status_code.sum(['service','status']).downsampling(SUM)
+```
+最后，我们便可以来到 SkyWalking UI 页面新建 Nginx 仪表板，使用刚刚 MAL 中定义的指标信息创建 Nginx Dashboard：
Review Comment:
Or a screenshot?
We planned to provide official Nginx dashboard,
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
kezhenxu94 commented on code in PR #652:
URL:
https://github.com/apache/skywalking-website/pull/652#discussion_r1381231878
##
content/zh/2023-10-29-collect-and-analyse-nginx-accesslog-by-lal/index.md:
##
@@ -0,0 +1,157 @@
+---
+title: "使用 LAL 收集并分析 Nginx access log"
+author: "魏翔"
+date: 2023-10-29
+description: "本篇文章演示如何将 Nginx access log 收集到 SkyWalking 中，并通过 LAL 进行指标分析"
+tags:
+- LAL
+- Logging
+- Nginx
+---
+
+## 背景介绍
+Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access
log，我们可以实现对 Nginx 中接口状态的监控。
+
+在本案例中，将由 [fluent-bit](https://fluentbit.io/) 收集 access log，并通过 HTTP 将日志信息发送给
SkyWalking OAP Server 进行进一步的分析。
+
+## 环境准备
+实验需要的 Nginx 及 Fluent-bit
相关配置文件都被上传到了[Github](https://github.com/weixiang1862/nginx-fluent-bit)，有需要的读者可以自行
git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。
+
+### Nginx日志格式配置
+LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.
+
+```
+http {
+...
+...
+
+log_format main '{"remote_addr": "$remote_addr",'
+'"remote_user": "$remote_user",'
+'"request": "$request",'
+'"time": "$time_iso8601",'
+'"status": "$status",'
+'"request_time":"$request_time",'
+'"body_bytes_sent": "$body_bytes_sent",'
+'"http_referer": "$http_referer",'
+'"http_user_agent": "$http_user_agent",'
+'"http_x_forwarded_for": "$http_x_forwarded_for"}';
+
+access_log /var/log/nginx/access.log main;
+
+...
+...
+}
+```
+
+### Fluent bit Filter
+我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：
+- body：日志内容体
+- service：服务名称
+- serviceInstance：实例名称
+```
+function rewrite_body(tag, timestamp, record)
+local newRecord = {}
+newRecord["body"] = { json = { json = record.log } }
+newRecord["service"] = "nginx::nginx"
+newRecord["serviceInstance"] = "localhost"
+return 1, timestamp, newRecord
+end
+```
+
+## OAP 日志分析
+### LAL定义
+在 filter 中，我们通过条件判断，只处理 `service=nginx::nginx` 的服务，其他服务依旧走默认逻辑：
+
+第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。
+
+第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log
json 中的 time。
+
+第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。
+
+第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：
+- `nginx_log_count`：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。
+- `nginx_request_time`：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。
+- `nginx_body_bytes_sent`：body 大小指标可以帮助我们了解网关上的流量情况。
+- `nginx_status_code`：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。
+
+```yaml
+rules:
+ - name: default
+layer: GENERAL
+dsl: |
+ filter {
+if (log.service == "nginx::nginx") {
+ json {
+abortOnFailure true
+ }
+
+ extractor {
+timestamp parsed.time as String, "-MM-dd'T'HH:mm:ssXXX"
+tag status: parsed.status
+tag remote_addr: parsed.remote_addr
+
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_log_count"
+ value 1
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_request_time"
+ value parsed.request_time as Double
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance
+ name "nginx_body_bytes_sent"
+ value parsed.body_bytes_sent as Long
+}
+metrics {
+ timestamp log.timestamp as Long
+ labels service: log.service, instance: log.serviceInstance,
status: parsed.status
+ name "nginx_status_code"
+ value 1
+}
+ }
+}
+
+sink {
+}
+ }
+```
+经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL 中提取的指标进行进一步分析：
+
+
+
+### MAL定义
+在 MAL 中，我们可以对上一步 LAL 中提取的指标进行进一步的分析聚合，下面的例子里：
+
+nginx_log_count、nginx_request_time、nginx_status_code 使用 sum 聚合函数处理，并使用 SUM 方式
downsampling，
+
+nginx_request_time 使用 avg 聚合函数求平均值，默认使用 AVG 方式 downsampling。
+
+完成聚合分析后，SkyWalking Meter System 会完成对上述指标的持久化。
+
+```yaml
+expSuffix: service(['service'], Layer.GENERAL)
+metricPrefix: nginx
+metricsRules:
+ - name: cpm
+exp: nginx_log_count.sum(['service']).downsampling(SUM)
+ - name: avg_request_time
+exp: nginx_request_time.avg(['service'])
+ - name: body_bytes_sent_count
+exp: nginx_body_bytes_sent.sum(['service']).downsampling(SUM)
+ - name: status_code_count
+exp: nginx_status_code.sum(['service','status']).downsampling(SUM)
+```
+最后，我们便可以来到 SkyWalking UI 页面新建 Nginx 仪表板，使用刚刚 MAL 中定义的指标信息创建 Nginx Dashboard：
Review Comment:
It'd be very helpful if you can also share the sample dashboard json
Re: [PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
netlify[bot] commented on PR #652: URL: https://github.com/apache/skywalking-website/pull/652#issuecomment-1791961922 ### Deploy Preview for *skywalking-website-preview* processing. | Name | Link | |:-:|| | Latest commit | d9fccd6c006b2b165a486fa84181593afc278cde | | Latest deploy log | https://app.netlify.com/sites/skywalking-website-preview/deploys/654499b255b5d200079005d2 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Add blog: collect and analyse nginx access log by LAL. [skywalking-website]
weixiang1862 opened a new pull request, #652: URL: https://github.com/apache/skywalking-website/pull/652 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support extract timestamp from patterned datetime string in LAL. [skywalking]
wu-sheng merged PR #11489: URL: https://github.com/apache/skywalking/pull/11489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking) branch master updated: Support extract timestamp from patterned datetime string in LAL. (#11489)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new 0f8c67822d Support extract timestamp from patterned datetime string in
LAL. (#11489)
0f8c67822d is described below
commit 0f8c67822d9e07cc446e66579c57cfcb2e035297
Author: weixiang1862 <652048...@qq.com>
AuthorDate: Fri Nov 3 14:42:04 2023 +0800
Support extract timestamp from patterned datetime string in LAL. (#11489)
---
docs/en/changes/changes.md | 1 +
docs/en/concepts-and-designs/lal.md| 21 ++-
.../analyzer/dsl/spec/extractor/ExtractorSpec.java | 25 --
.../skywalking/oap/log/analyzer/dsl/DSLTest.java | 11
.../query/graphql/resolver/LogTestQueryTest.java | 30 ++
test/e2e-v2/cases/vm/zabbix/docker-compose.yml | 2 +-
6 files changed, 86 insertions(+), 4 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index e464311ec2..893f5f4ae8 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -35,6 +35,7 @@
* BanyanDBStorageClient: Add `define(Property property, PropertyStore.Strategy
strategy)` API.
* Support GraalVM native-image (Experimental).
* Correct the file format and fix typos in the filenames for monitoring
Kafka's e2e tests.
+* Support extract timestamp from patterned datetime string in LAL.
UI
diff --git a/docs/en/concepts-and-designs/lal.md
b/docs/en/concepts-and-designs/lal.md
index f81d96f125..caa934a923 100644
--- a/docs/en/concepts-and-designs/lal.md
+++ b/docs/en/concepts-and-designs/lal.md
@@ -185,7 +185,26 @@ dropped) and is used to associate with traces / metrics.
`timestamp` extracts the timestamp from the `parsed` result, and set it into
the `LogData`, which will be persisted (if
not dropped) and is used to associate with traces / metrics.
-The unit of `timestamp` is millisecond.
+The parameter of `timestamp` can be a millisecond:
+```groovy
+filter {
+// ... parser
+
+extractor {
+timestamp parsed.time as String
+}
+}
+```
+or a datetime string with a specified pattern:
+```groovy
+filter {
+// ... parser
+
+extractor {
+timestamp parsed.time as String, "-MM-dd HH:mm:ss"
+}
+}
+```
- `layer`
diff --git
a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java
b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java
index 9359fff070..2a52cd64e1 100644
---
a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java
+++
b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java
@@ -22,6 +22,8 @@ import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableMap;
import groovy.lang.Closure;
import groovy.lang.DelegatesTo;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
import java.util.Collection;
import java.util.List;
import java.util.Map;
@@ -57,6 +59,7 @@ import
org.apache.skywalking.oap.server.core.source.SourceReceiver;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.library.module.ModuleStartException;
import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -203,11 +206,29 @@ public class ExtractorSpec extends AbstractSpec {
@SuppressWarnings("unused")
public void timestamp(final String timestamp) {
+timestamp(timestamp, null);
+}
+
+@SuppressWarnings("unused")
+public void timestamp(final String timestamp, final String formatPattern) {
if (BINDING.get().shouldAbort()) {
return;
}
-if (nonNull(timestamp) && StringUtils.isNumeric(timestamp)) {
-BINDING.get().log().setTimestamp(Long.parseLong(timestamp));
+if (StringUtil.isEmpty(timestamp)) {
+return;
+}
+
+if (StringUtil.isEmpty(formatPattern)) {
+if (StringUtils.isNumeric(timestamp)) {
+BINDING.get().log().setTimestamp(Long.parseLong(timestamp));
+}
+} else {
+SimpleDateFormat format = new SimpleDateFormat(formatPattern);
+try {
+
BINDING.get().log().setTimestamp(format.parse(timestamp).getTime());
+} catch (ParseException e) {
+// ignore
+}
}
}
diff --git
a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java
Re: [I] [Bug] PHP agent don't have any endpoint info [skywalking]
aclstack commented on issue #11488:
URL: https://github.com/apache/skywalking/issues/11488#issuecomment-1791948609
when set skywalking_agent.log_level=TRACE and restart demo , i access
http://127.0.0.1:/ the log info is
```
2023-11-03T06:26:49.006925Z INFO skywalking_agent::worker: Start to
shutdown skywalking grpc reporter
2023-11-03T06:26:49.006978Z INFO skywalking_agent::worker: Remove socket
file socket_file="/tmp/skywalking-agent/609399453fa45.sock"
2023-11-03T06:26:51.447624Z INFO skywalking_agent::module: Starting
skywalking agent service_name="hello-skywalking"
service_instance="69498981748909088254532449806842367552@172.17.0.4"
skywalking_version=9 heartbeat_period=30 properties_report_period_factor=10
2023-11-03T06:26:51.452122Z DEBUG skywalking_agent::worker: Starting
worker...
2023-11-03T06:26:51.452258Z DEBUG skywalking_agent::worker: Bind unix stream
socket_file="/tmp/skywalking-agent/6093997008c52.sock"
2023-11-03T06:26:51.452446Z DEBUG skywalking_agent::reporter::reporter_grpc:
Create Endpoint url="http://192.168.9.247:11800;
2023-11-03T06:26:51.452480Z DEBUG skywalking_agent::reporter::reporter_grpc:
Skywalking TLS info enable_tls=false ssl_trusted_ca_path="" ssl_key_path=""
ssl_cert_chain_path=""
2023-11-03T06:26:51.454907Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.454946Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("define")
class_name=None
2023-11-03T06:26:51.455129Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.455476Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.455750Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.455870Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.455897Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.456563Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.45Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.456749Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.456953Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.456966Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("method_exists")
class_name=None
2023-11-03T06:26:51.456981Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("define")
class_name=None
2023-11-03T06:26:51.457025Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.457109Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.457145Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.457629Z DEBUG skywalking_agent::worker: Report instance
properties props=Properties { inner: {"hostname": ["766b2c633708"], "ipv4":
["172.17.0.4"], "OS Name": ["Linux"], "language": ["php"], "Process No.":
["1495"]} }
2023-11-03T06:26:51.457680Z INFO connect:
skywalking_agent::reporter::reporter_grpc: Skywalking server connected
uri="http://192.168.9.247:11800/;
2023-11-03T06:26:51.457712Z INFO skywalking_agent::reporter::reporter_grpc:
Worker is ready...
2023-11-03T06:26:51.457831Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.457983Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.458015Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.458115Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.458184Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.458259Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.458445Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=None class_name=None
2023-11-03T06:26:51.458611Z TRACE
Re: [I] [Bug] PHP agent don't have any endpoint info [skywalking]
aclstack commented on issue #11488:
URL: https://github.com/apache/skywalking/issues/11488#issuecomment-1791942202
if i set skywalking_agent.log_level=TRACE when i access
http://127.0.0.1: the log info
```
2023-11-03T06:22:25.465807Z TRACE skywalking_agent::execute: execute_ex
function and class name function_name=Some("{closure}") class_name=None
2023-11-03T06:22:25.465863Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("header")
class_name=Some("Swoole\\Http\\Response")
2023-11-03T06:22:25.465877Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("rand")
class_name=None
2023-11-03T06:22:25.465885Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("end")
class_name=Some("Swoole\\Http\\Response")
2023-11-03T06:22:25.465948Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("__destruct")
class_name=Some("Swoole\\Http\\Request")
2023-11-03T06:22:25.465966Z TRACE skywalking_agent::execute:
execute_internal function and class name function_name=Some("__destruct")
class_name=Some("Swoole\\Http\\Response")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
