This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch docs in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit f582002a965c8832a7701284c5b9a79120842350 Author: Wu Sheng <[email protected]> AuthorDate: Thu Dec 15 15:13:02 2022 +0800 Reorganize API docs. --- README.md | 2 - docs/en/FAQ/v8-version-upgrade.md | 2 +- .../browser-http-api-protocol.md} | 2 +- .../browser-protocol.md} | 2 +- docs/en/api/event.md | 94 +++++++++++ docs/en/api/instance-properties.md | 93 ++++++++++ docs/en/api/jvm-protocol.md | 99 +++++++++++ docs/en/api/log-data-protocol.md | 187 +++++++++++++++++++++ docs/en/api/meter.md | 91 ++++++++++ docs/en/{protocols => api}/query-protocol.md | 0 .../trace-data-protocol-v3.md} | 158 +++++++++++++++-- .../x-process-correlation-headers-v1.md} | 14 +- .../x-process-propagation-headers-v3.md} | 0 docs/en/changes/changes.md | 1 + docs/en/concepts-and-designs/manual-sdk.md | 4 +- docs/en/protocols/HTTP-API-Protocol.md | 186 -------------------- docs/en/protocols/JVM-Protocol.md | 5 - docs/en/protocols/Log-Data-Protocol.md | 85 ---------- docs/en/protocols/README.md | 63 ------- docs/en/setup/backend/log-analyzer.md | 4 +- docs/en/setup/backend/mq.md | 2 +- docs/menu.yml | 34 +++- 22 files changed, 755 insertions(+), 373 deletions(-) diff --git a/README.md b/README.md index 4097ad8032..36916f375b 100644 --- a/README.md +++ b/README.md @@ -56,8 +56,6 @@ and multiple formats, including # Documentation - [Official documentation](https://skywalking.apache.org/docs/#SkyWalking) -NOTICE, SkyWalking 8.0+ uses [v3 protocols](docs/en/protocols/README.md). They are incompatible with previous releases. - SkyWalking OAP is using the STAM(Streaming Topology Analysis Method) to analysis topology in the tracing based agent scenario for better performance. Read [the paper of STAM](https://wu-sheng.github.io/STAM/) for more details. diff --git a/docs/en/FAQ/v8-version-upgrade.md b/docs/en/FAQ/v8-version-upgrade.md index df0d001407..5b84d66825 100644 --- a/docs/en/FAQ/v8-version-upgrade.md +++ b/docs/en/FAQ/v8-version-upgrade.md @@ -1,5 +1,5 @@ # V8 upgrade -Starting from SkyWalking v8, the [v3 protocol](../protocols/README.md) has been used. This makes it incompatible with previous releases. +Starting from SkyWalking v8, the [v3 protocol](../api/trace-data-protocol-v3.md) has been used. This makes it incompatible with previous releases. Users who intend to upgrade in v8 series releases could follow the steps below. diff --git a/docs/en/protocols/Browser-HTTP-API-Protocol.md b/docs/en/api/browser-http-api-protocol.md similarity index 97% rename from docs/en/protocols/Browser-HTTP-API-Protocol.md rename to docs/en/api/browser-http-api-protocol.md index 72265e002e..6966c21f5b 100644 --- a/docs/en/protocols/Browser-HTTP-API-Protocol.md +++ b/docs/en/api/browser-http-api-protocol.md @@ -1,7 +1,7 @@ # HTTP API Protocol HTTP API Protocol defines the API data format, including API request and response data format. -They use the HTTP1.1 wrapper of the official [SkyWalking Browser Protocol](Browser-Protocol.md). Read it for more details. +They use the HTTP1.1 wrapper of the official [SkyWalking Browser Protocol](browser-protocol.md). Read it for more details. ## Performance Data Report diff --git a/docs/en/protocols/Browser-Protocol.md b/docs/en/api/browser-protocol.md similarity index 91% rename from docs/en/protocols/Browser-Protocol.md rename to docs/en/api/browser-protocol.md index 44607fb076..1a57f11bac 100644 --- a/docs/en/protocols/Browser-Protocol.md +++ b/docs/en/api/browser-protocol.md @@ -5,7 +5,7 @@ Browser protocol describes the data format between [skywalking-client-js](https: ## Overview Browser protocol is defined and provided in [gRPC format](https://github.com/apache/skywalking-data-collect-protocol/blob/master/browser/BrowserPerf.proto), -and also implemented in [HTTP 1.1](Browser-HTTP-API-Protocol.md) +and also implemented in [HTTP 1.1](browser-http-api-protocol.md) ### Send performance data and error logs diff --git a/docs/en/api/event.md b/docs/en/api/event.md new file mode 100644 index 0000000000..a81210c092 --- /dev/null +++ b/docs/en/api/event.md @@ -0,0 +1,94 @@ +### Events Report Protocol + +The protocol is used to report events to the backend. The [doc](../concepts-and-designs/event.md) introduces the definition of an event, and [the protocol repository](https://github.com/apache/skywalking-data-collect-protocol/blob/master/event) defines gRPC services and message formats of events. + +```protobuf +syntax = "proto3"; + +package skywalking.v3; + +option java_multiple_files = true; +option java_package = "org.apache.skywalking.apm.network.event.v3"; +option csharp_namespace = "SkyWalking.NetworkProtocol.V3"; +option go_package = "skywalking.apache.org/repo/goapi/collect/event/v3"; + +import "common/Command.proto"; + +service EventService { + // When reporting an event, you typically call the collect function twice, one for starting of the event and the other one for ending of the event, with the same UUID. + // There are also cases where you have both start time and end time already, for example, when exporting events from a 3rd-party system, + // the start time and end time are already known so that you can call the collect function only once. + rpc collect (stream Event) returns (Commands) { + } +} + +message Event { + // Unique ID of the event. Because an event may span a long period of time, the UUID is necessary to associate the + // start time with the end time of the same event. + string uuid = 1; + + // The source object that the event occurs on. + Source source = 2; + + // The name of the event. For example, `Reboot`, `Upgrade` etc. + string name = 3; + + // The type of the event. This field is friendly for UI visualization, where events of type `Normal` are considered as normal operations, + // while `Error` is considered as unexpected operations, such as `Crash` events, therefore we can mark them with different colors to be easier identified. + Type type = 4; + + // The detail of the event that describes why this event happened. This should be a one-line message that briefly describes why the event is reported. + // Examples of an `Upgrade` event may be something like `Upgrade from ${from_version} to ${to_version}`. + // It's NOT encouraged to include the detailed logs of this event, such as the exception stack trace. + string message = 5; + + // The parameters in the `message` field. + map<string, string> parameters = 6; + + // The start time (in milliseconds) of the event, measured between the current time and midnight, January 1, 1970 UTC. + // This field is mandatory when an event occurs. + int64 startTime = 7; + + // The end time (in milliseconds) of the event. , measured between the current time and midnight, January 1, 1970 UTC. + // This field may be empty if the event has not stopped yet, otherwise it should be a valid timestamp after `startTime`. + int64 endTime = 8; + + // [Required] Since 9.0.0 + // Name of the layer to which the event belongs. + string layer = 9; +} + +enum Type { + Normal = 0; + Error = 1; +} + +// If the event occurs on a service ONLY, the `service` field is mandatory, the serviceInstance field and endpoint field are optional; +// If the event occurs on a service instance, the `service` and `serviceInstance` are mandatory and endpoint is optional; +// If the event occurs on an endpoint, `service` and `endpoint` are mandatory, `serviceInstance` is optional; +message Source { + string service = 1; + string serviceInstance = 2; + string endpoint = 3; +} +``` + +`JSON` format events can be reported via HTTP API. The endpoint is `http://<oap-address>:12800/v3/events`. +Example of a JSON event record: +```json +[ + { + "uuid": "f498b3c0-8bca-438d-a5b0-3701826ae21c", + "source": { + "service": "SERVICE-A", + "instance": "INSTANCE-1" + }, + "name": "Reboot", + "type": "Normal", + "message": "App reboot.", + "parameters": {}, + "startTime": 1628044330000, + "endTime": 1628044331000 + } +] +``` \ No newline at end of file diff --git a/docs/en/api/instance-properties.md b/docs/en/api/instance-properties.md new file mode 100644 index 0000000000..e0212b0965 --- /dev/null +++ b/docs/en/api/instance-properties.md @@ -0,0 +1,93 @@ +# Report service instance status +1. Service Instance Properties + Service instance contains more information than just a name. In order for the agent to report service instance status, use `ManagementService#reportInstanceProperties` service to provide a string-key/string-value pair list as the parameter. The `language` of target instance must be provided as the minimum requirement. + +2. Service Ping + Service instance should keep alive with the backend. The agent should set a scheduler using `ManagementService#keepAlive` service every minute. + + +```protobuf +syntax = "proto3"; + +package skywalking.v3; + +option java_multiple_files = true; +option java_package = "org.apache.skywalking.apm.network.management.v3"; +option csharp_namespace = "SkyWalking.NetworkProtocol.V3"; +option go_package = "skywalking.apache.org/repo/goapi/collect/management/v3"; + +import "common/Common.proto"; +import "common/Command.proto"; + +// Define the service reporting the extra information of the instance. +service ManagementService { + // Report custom properties of a service instance. + rpc reportInstanceProperties (InstanceProperties) returns (Commands) { + } + + // Keep the instance alive in the backend analysis. + // Only recommend to do separate keepAlive report when no trace and metrics needs to be reported. + // Otherwise, it is duplicated. + rpc keepAlive (InstancePingPkg) returns (Commands) { + + } +} + +message InstanceProperties { + string service = 1; + string serviceInstance = 2; + repeated KeyStringValuePair properties = 3; + // Instance belong layer name which define in the backend, general is default. + string layer = 4; +} + +message InstancePingPkg { + string service = 1; + string serviceInstance = 2; + // Instance belong layer name which define in the backend, general is default. + string layer = 3; +} +``` + +## Via HTTP Endpoint + +- Report service instance properties + +> POST http://localhost:12800/v3/management/reportProperties + +Input: + +```json +{ + "service": "User Service Name", + "serviceInstance": "User Service Instance Name", + "properties": [ + { "key": "language", "value": "Lua" } + ] +} +``` + +Output JSON Array: + +```json +{} +``` + +- Service instance ping + +> POST http://localhost:12800/v3/management/keepAlive + +Input: + +```json +{ + "service": "User Service Name", + "serviceInstance": "User Service Instance Name" +} +``` + +OutPut: + +```json +{} +``` \ No newline at end of file diff --git a/docs/en/api/jvm-protocol.md b/docs/en/api/jvm-protocol.md new file mode 100644 index 0000000000..2374a8d3a6 --- /dev/null +++ b/docs/en/api/jvm-protocol.md @@ -0,0 +1,99 @@ +# JVM Metrics APIs + +**Notice, SkyWalking has provided general available [meter APIs](meter.md) for all kinds of metrics. This API is still supported +for forward compatibility only. SkyWalking community would not accept new language specific metric APIs anymore.** + +Uplink the JVM metrics, including PermSize, HeapSize, CPU, Memory, etc., every second. + +[gRPC service define](https://github.com/apache/skywalking-data-collect-protocol/blob/master/language-agent/JVMMetric.proto) + +```protobuf +syntax = "proto3"; + +package skywalking.v3; + +option java_multiple_files = true; +option java_package = "org.apache.skywalking.apm.network.language.agent.v3"; +option csharp_namespace = "SkyWalking.NetworkProtocol.V3"; +option go_package = "skywalking.apache.org/repo/goapi/collect/language/agent/v3"; + +import "common/Common.proto"; +import "common/Command.proto"; + +// Define the JVM metrics report service. +service JVMMetricReportService { + rpc collect (JVMMetricCollection) returns (Commands) { + } +} + +message JVMMetricCollection { + repeated JVMMetric metrics = 1; + string service = 2; + string serviceInstance = 3; +} + +message JVMMetric { + int64 time = 1; + CPU cpu = 2; + repeated Memory memory = 3; + repeated MemoryPool memoryPool = 4; + repeated GC gc = 5; + Thread thread = 6; + Class clazz = 7; +} + +message Memory { + bool isHeap = 1; + int64 init = 2; + int64 max = 3; + int64 used = 4; + int64 committed = 5; +} + +message MemoryPool { + PoolType type = 1; + int64 init = 2; + int64 max = 3; + int64 used = 4; + int64 committed = 5; +} + +enum PoolType { + CODE_CACHE_USAGE = 0; + NEWGEN_USAGE = 1; + OLDGEN_USAGE = 2; + SURVIVOR_USAGE = 3; + PERMGEN_USAGE = 4; + METASPACE_USAGE = 5; +} + +message GC { + GCPhase phase = 1; + int64 count = 2; + int64 time = 3; +} + +enum GCPhase { + NEW = 0; + OLD = 1; + NORMAL = 2; // The type of GC doesn't have new and old phases, like Z Garbage Collector (ZGC) +} + +// See: https://docs.oracle.com/javase/8/docs/api/java/lang/management/ThreadMXBean.html +message Thread { + int64 liveCount = 1; + int64 daemonCount = 2; + int64 peakCount = 3; + int64 runnableStateThreadCount = 4; + int64 blockedStateThreadCount = 5; + int64 waitingStateThreadCount = 6; + int64 timedWaitingStateThreadCount = 7; +} + +// See: https://docs.oracle.com/javase/8/docs/api/java/lang/management/ClassLoadingMXBean.html +message Class { + int64 loadedClassCount = 1; + int64 totalUnloadedClassCount = 2; + int64 totalLoadedClassCount = 3; +} +``` diff --git a/docs/en/api/log-data-protocol.md b/docs/en/api/log-data-protocol.md new file mode 100644 index 0000000000..a2ffbe1edd --- /dev/null +++ b/docs/en/api/log-data-protocol.md @@ -0,0 +1,187 @@ +# Log Data Protocol + +Report log data via protocol. + +## Native Proto Protocol + +Report `native-proto` format log via gRPC. + +[gRPC service define](https://github.com/apache/skywalking-data-collect-protocol/blob/master/logging/Logging.proto) + +```protobuf +syntax = "proto3"; + +package skywalking.v3; + +option java_multiple_files = true; +option java_package = "org.apache.skywalking.apm.network.logging.v3"; +option csharp_namespace = "SkyWalking.NetworkProtocol.V3"; +option go_package = "skywalking.apache.org/repo/goapi/collect/logging/v3"; + +import "common/Common.proto"; +import "common/Command.proto"; + +// Report collected logs into the OAP backend +service LogReportService { + // Recommend to report log data in a stream mode. + // The service/instance/endpoint of the log could share the previous value if they are not set. + // Reporting the logs of same service in the batch mode could reduce the network cost. + rpc collect (stream LogData) returns (Commands) { + } +} + +// Log data is collected through file scratcher of agent. +// Natively, Satellite provides various ways to collect logs. +message LogData { + // [Optional] The timestamp of the log, in millisecond. + // If not set, OAP server would use the received timestamp as log's timestamp, or relies on the OAP server analyzer. + int64 timestamp = 1; + // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests. + // + // The logic name represents the service. This would show as a separate node in the topology. + // The metrics analyzed from the spans, would be aggregated for this entity as the service level. + // + // If this is not the first element of the streaming, use the previous not-null name as the service name. + string service = 2; + // [Optional] **Service Instance**. Each individual workload in the Service group is known as an instance. Like `pods` in Kubernetes, it + // doesn't need to be a single OS process, however, if you are using instrument agents, an instance is actually a real OS process. + // + // The logic name represents the service instance. This would show as a separate node in the instance relationship. + // The metrics analyzed from the spans, would be aggregated for this entity as the service instance level. + string serviceInstance = 3; + // [Optional] **Endpoint**. A path in a service for incoming requests, such as an HTTP URI path or a gRPC service class + method signature. + // + // The logic name represents the endpoint, which logs belong. + string endpoint = 4; + // [Required] The content of the log. + LogDataBody body = 5; + // [Optional] Logs with trace context + TraceContext traceContext = 6; + // [Optional] The available tags. OAP server could provide search/analysis capabilities based on these. + LogTags tags = 7; + // [Optional] Since 9.0.0 + // The layer of the service and servce instance. If absent, the OAP would set `layer`=`ID: 2, NAME: general` + string layer = 8; +} + +// The content of the log data +message LogDataBody { + // A type to match analyzer(s) at the OAP server. + // The data could be analyzed at the client side, but could be partial + string type = 1; + // Content with extendable format. + oneof content { + TextLog text = 2; + JSONLog json = 3; + YAMLLog yaml = 4; + } +} + +// Literal text log, typically requires regex or split mechanism to filter meaningful info. +message TextLog { + string text = 1; +} + +// JSON formatted log. The json field represents the string that could be formatted as a JSON object. +message JSONLog { + string json = 1; +} + +// YAML formatted log. The yaml field represents the string that could be formatted as a YAML map. +message YAMLLog { + string yaml = 1; +} + +// Logs with trace context, represent agent system has injects context(IDs) into log text. +message TraceContext { + // [Optional] A string id represents the whole trace. + string traceId = 1; + // [Optional] A unique id represents this segment. Other segments could use this id to reference as a child segment. + string traceSegmentId = 2; + // [Optional] The number id of the span. Should be unique in the whole segment. + // Starting at 0. + int32 spanId = 3; +} + +message LogTags { + // String key, String value pair. + repeated KeyStringValuePair data = 1; +} + +``` + +## Native Kafka Protocol + +Report `native-json` format log via kafka. + +Json log record example: +```json +{ + "timestamp":1618161813371, + "service":"Your_ApplicationName", + "serviceInstance":"[email protected]", + "layer":"GENERAL", + "traceContext":{ + "traceId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001", + "spanId":"0", + "traceSegmentId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000" + }, + "tags":{ + "data":[ + { + "key":"level", + "value":"INFO" + }, + { + "key":"logger", + "value":"com.example.MyLogger" + } + ] + }, + "body":{ + "text":{ + "text":"log message" + } + } +} +``` + +## HTTP API + +Report `json` format logs via HTTP API, the endpoint is `http://<oap-address>:12800/v3/logs`. + +Json log record example: + +```json +[ + { + "timestamp": 1618161813371, + "service": "Your_ApplicationName", + "serviceInstance": "[email protected]", + "layer":"GENERAL", + "traceContext": { + "traceId": "ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001", + "spanId": "0", + "traceSegmentId": "ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000" + }, + "tags": { + "data": [ + { + "key": "level", + "value": "INFO" + }, + { + "key": "logger", + "value": "com.example.MyLogger" + } + ] + }, + "body": { + "text": { + "text": "log message" + } + } + } +] +``` + diff --git a/docs/en/api/meter.md b/docs/en/api/meter.md new file mode 100644 index 0000000000..e6f7f44c7e --- /dev/null +++ b/docs/en/api/meter.md @@ -0,0 +1,91 @@ +# Meter APIs +SkyWalking has a native metrics format, and supports widely used metric formats, such as Prometheus, OpenCensus, OpenTelemetry, and Zabbix. + +```protobuf +syntax = "proto3"; + +package skywalking.v3; + +option java_multiple_files = true; +option java_package = "org.apache.skywalking.apm.network.language.agent.v3"; +option go_package = "skywalking.apache.org/repo/goapi/collect/language/agent/v3"; + +import "common/Command.proto"; + +service MeterReportService { + // Meter data is reported in a certain period. The agent/SDK should report all collected metrics in this period through one stream. + // The whole stream is an input data set, client should onComplete the stream per report period. + rpc collect (stream MeterData) returns (Commands) { + } + + // Reporting meter data in bulk mode as MeterDataCollection. + // By using this, each one in the stream would be treated as a complete input for MAL engine, + // comparing to `collect (stream MeterData)`, which is using one stream as an input data set. + rpc collectBatch (stream MeterDataCollection) returns (Commands) { + } +} + +// Label of the meter +message Label { + string name = 1; + string value = 2; +} + +// The histogram element definition. It includes the bucket lower boundary and the count in the bucket. +message MeterBucketValue { + // The value represents the min value of the bucket, + // the upper boundary is determined by next MeterBucketValue$bucket, + // if it doesn't exist, the upper boundary is positive infinity. + double bucket = 1; + int64 count = 2; + // If is negative infinity, the value of the bucket is invalid + bool isNegativeInfinity = 3; +} + +// Meter single value +message MeterSingleValue { + // Meter name + string name = 1; + // Labels + repeated Label labels = 2; + // Single value + double value = 3; +} + +// Histogram +message MeterHistogram { + // Meter name + string name = 1; + // Labels + repeated Label labels = 2; + // Customize the buckets + repeated MeterBucketValue values = 3; +} + +// Single meter data, if the same metrics have a different label, they will separate. +message MeterData { + // Meter data could be a single value or histogram. + oneof metric { + MeterSingleValue singleValue = 1; + MeterHistogram histogram = 2; + } + // Service name, be set value in the first element in the stream-call. + string service = 3; + // Service instance name, be set value in the first element in the stream-call. + string serviceInstance = 4; + // Meter data report time, be set value in the first element in the stream-call. + int64 timestamp = 5; +} + +message MeterDataCollection { + repeated MeterData meterData = 1; +} +``` + +OpenTelemetry collector, Telegraf agents, Zabbix agents could use their native protocol(e.g. OTLP) +and OAP server would convert metrics into native format and forward them to [Meter Analysis Language](../concepts-and-designs/mal.md) engine. + +To learn more about receiving 3rd party formats metrics, see +- [Meter receiver](../setup/backend/backend-meter.md) +- [OpenTelemetry receiver](../setup/backend/opentelemetry-receiver.md). +- [Zabbix receiver](../setup/backend/backend-zabbix.md) diff --git a/docs/en/protocols/query-protocol.md b/docs/en/api/query-protocol.md similarity index 100% rename from docs/en/protocols/query-protocol.md rename to docs/en/api/query-protocol.md diff --git a/docs/en/protocols/Trace-Data-Protocol-v3.md b/docs/en/api/trace-data-protocol-v3.md similarity index 81% rename from docs/en/protocols/Trace-Data-Protocol-v3.md rename to docs/en/api/trace-data-protocol-v3.md index 5d2fddc7e8..1b71155851 100644 --- a/docs/en/protocols/Trace-Data-Protocol-v3.md +++ b/docs/en/api/trace-data-protocol-v3.md @@ -1,20 +1,9 @@ -# Trace Data Protocol v3.1 -Trace Data Protocol describes the data format between SkyWalking agent/sniffer and backend. - -Trace data protocol is defined and provided in [gRPC format](https://github.com/apache/skywalking-data-collect-protocol), and implemented in [HTTP 1.1](HTTP-API-Protocol.md). - -## Report service instance status -1. Service Instance Properties -Service instance contains more information than just a name. In order for the agent to report service instance status, use `ManagementService#reportInstanceProperties` service to provide a string-key/string-value pair list as the parameter. The `language` of target instance must be provided as the minimum requirement. +# Trace Data Protocol +* Version, v3.1 -2. Service Ping -Service instance should keep alive with the backend. The agent should set a scheduler using `ManagementService#keepAlive` service every minute. +Trace Data Protocol describes the data format between SkyWalking agent/sniffer and backend. -## Send trace and JVM metrics -After you have the service ID and service instance ID ready, you could send traces and metrics. Now we -have -1. `TraceSegmentReportService#collect` for the SkyWalking native trace format -1. `JVMMetricReportService#collect` for the SkyWalking native JVM format +Trace data protocol is defined and provided in [gRPC format](https://github.com/apache/skywalking-data-collect-protocol), and implemented in HTTP 1.1. For trace format, note that: 1. The segment is a unique concept in SkyWalking. It should include all spans for each request in a single OS process, which is usually a single language-based thread. @@ -35,7 +24,7 @@ For example, accessing DB by JDBC, and reading Redis/Memcached are classified as 3. Cross-thread/process span parent information is called "reference". Reference carries the trace ID, segment ID, span ID, service name, service instance name, endpoint name, and target address used on the client end (note: this is not required in cross-thread operations) of this request in the parent. -See [Cross Process Propagation Headers Protocol v3](Skywalking-Cross-Process-Propagation-Headers-Protocol-v3.md) for more details. +See [Cross Process Propagation Headers Protocol v3](x-process-propagation-headers-v3.md) for more details. 4. `Span#skipAnalysis` may be TRUE, if this span doesn't require backend analysis. @@ -285,4 +274,141 @@ message SpanAttachedEvent { ZIPKIN = 1; } } +``` + +## Via HTTP Endpoint + +Detailed information about data format can be found in [Instance Management](https://github.com/apache/skywalking-data-collect-protocol/blob/master/language-agent/Tracing.proto). +There are two ways to report segment data: one segment per request or segment array in bulk mode. + +### POST http://localhost:12800/v3/segment + +Send a single segment object in JSON format. + +Input: + +```json +{ + "traceId": "a12ff60b-5807-463b-a1f8-fb1c8608219e", + "serviceInstance": "User_Service_Instance_Name", + "spans": [{ + "operationName": "/ingress", + "startTime": 1588664577013, + "endTime": 1588664577028, + "spanType": "Exit", + "spanId": 1, + "isError": false, + "parentSpanId": 0, + "componentId": 6000, + "peer": "upstream service", + "spanLayer": "Http" + }, { + "operationName": "/ingress", + "startTime": 1588664577013, + "tags": [{ + "key": "http.method", + "value": "GET" + }, { + "key": "http.params", + "value": "http://localhost/ingress"; + }], + "endTime": 1588664577028, + "spanType": "Entry", + "spanId": 0, + "parentSpanId": -1, + "isError": false, + "spanLayer": "Http", + "componentId": 6000 + }], + "service": "User_Service_Name", + "traceSegmentId": "a12ff60b-5807-463b-a1f8-fb1c8608219e" +} +``` +OutPut: + + ```json + +``` + +### POST http://localhost:12800/v3/segments + +Send a segment object list in JSON format. + +Input: + +```json +[{ + "traceId": "a12ff60b-5807-463b-a1f8-fb1c8608219e", + "serviceInstance": "User_Service_Instance_Name", + "spans": [{ + "operationName": "/ingress", + "startTime": 1588664577013, + "endTime": 1588664577028, + "spanType": "Exit", + "spanId": 1, + "isError": false, + "parentSpanId": 0, + "componentId": 6000, + "peer": "upstream service", + "spanLayer": "Http" + }, { + "operationName": "/ingress", + "startTime": 1588664577013, + "tags": [{ + "key": "http.method", + "value": "GET" + }, { + "key": "http.params", + "value": "http://localhost/ingress"; + }], + "endTime": 1588664577028, + "spanType": "Entry", + "spanId": 0, + "parentSpanId": -1, + "isError": false, + "spanLayer": "Http", + "componentId": 6000 + }], + "service": "User_Service_Name", + "traceSegmentId": "a12ff60b-5807-463b-a1f8-fb1c8608219e" +}, { + "traceId": "f956699e-5106-4ea3-95e5-da748c55bac1", + "serviceInstance": "User_Service_Instance_Name", + "spans": [{ + "operationName": "/ingress", + "startTime": 1588664577250, + "endTime": 1588664577250, + "spanType": "Exit", + "spanId": 1, + "isError": false, + "parentSpanId": 0, + "componentId": 6000, + "peer": "upstream service", + "spanLayer": "Http" + }, { + "operationName": "/ingress", + "startTime": 1588664577250, + "tags": [{ + "key": "http.method", + "value": "GET" + }, { + "key": "http.params", + "value": "http://localhost/ingress"; + }], + "endTime": 1588664577250, + "spanType": "Entry", + "spanId": 0, + "parentSpanId": -1, + "isError": false, + "spanLayer": "Http", + "componentId": 6000 + }], + "service": "User_Service_Name", + "traceSegmentId": "f956699e-5106-4ea3-95e5-da748c55bac1" +}] +``` +OutPut: + + ```json + ``` \ No newline at end of file diff --git a/docs/en/protocols/Skywalking-Cross-Process-Correlation-Headers-Protocol-v1.md b/docs/en/api/x-process-correlation-headers-v1.md similarity index 56% rename from docs/en/protocols/Skywalking-Cross-Process-Correlation-Headers-Protocol-v1.md rename to docs/en/api/x-process-correlation-headers-v1.md index 5bdf1de7fa..79c2be1d91 100644 --- a/docs/en/protocols/Skywalking-Cross-Process-Correlation-Headers-Protocol-v1.md +++ b/docs/en/api/x-process-correlation-headers-v1.md @@ -1,15 +1,21 @@ # SkyWalking Cross Process Correlation Headers Protocol + * Version 1.0 -The Cross Process Correlation Headers Protocol is used to transport custom data by leveraging the capability of [Cross Process Propagation Headers Protocol](Skywalking-Cross-Process-Propagation-Headers-Protocol-v3.md). +SkyWalking Cross Process Correlation Headers Protocol is a new in-wire context propagation protocol which is +additional and optional. Please read SkyWalking language agents documentation to see whether it is supported. -This is an optional and additional protocol for language tracer implementation. All tracer implementation could consider implementing this. -Cross Process Correlation Header key is `sw8-correlation`. The value is the `encoded(key):encoded(value)` list with elements splitted by `,` such as `base64(string key):base64(string value),base64(string key2):base64(string value2)`. +This is an optional and additional protocol for language tracer implementation. All tracer implementation could consider +implementing this. +Cross Process Correlation Header key is `sw8-correlation`. The value is the `encoded(key):encoded(value)` list with +elements splitted by `,` such as `base64(string key):base64(string value),base64(string key2):base64(string value2)`. ## Recommendations for language APIs + The following implementation method is recommended for different language APIs. -1. `TraceContext#putCorrelation` and `TraceContext#getCorrelation` are recommended to write and read the correlation context, with key/value string. +1. `TraceContext#putCorrelation` and `TraceContext#getCorrelation` are recommended to write and read the correlation + context, with key/value string. 1. The key should be added if it is absent. 1. The latter writes should override the previous value. 1. The total number of all keys should be less than 3, and the length of each value should be less than 128 bytes. diff --git a/docs/en/protocols/Skywalking-Cross-Process-Propagation-Headers-Protocol-v3.md b/docs/en/api/x-process-propagation-headers-v3.md similarity index 100% rename from docs/en/protocols/Skywalking-Cross-Process-Propagation-Headers-Protocol-v3.md rename to docs/en/api/x-process-propagation-headers-v3.md diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 015e71e9ac..9561c4792c 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -48,5 +48,6 @@ * Remove Spring Sleuth docs, and add `Spring MicroMeter Observations Analysis` with the latest Java agent side enhancement. * Update `monitoring MySQL document` to add the `MariaDB` part. +* Reorganize the protocols docs to a more clear API docs. All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/160?closed=1) diff --git a/docs/en/concepts-and-designs/manual-sdk.md b/docs/en/concepts-and-designs/manual-sdk.md index 0d231db704..498e5e2a54 100644 --- a/docs/en/concepts-and-designs/manual-sdk.md +++ b/docs/en/concepts-and-designs/manual-sdk.md @@ -5,7 +5,9 @@ Our incredible community has contributed to the manual instrument SDK. - [C++](https://github.com/SkyAPM/cpp2sky). C++ SDK follows the SkyWalking format. ## What are the SkyWalking format and the propagation protocols? -See these protocols in [protocols document](../protocols/README.md). +- [Tracing APIs](../api/trace-data-protocol-v3.md) +- [Meter APIs](../api/meter.md) +- [Logging APIs](../api/log-data-protocol.md) ## Envoy tracer Envoy has its internal tracer implementation for SkyWalking. Read [SkyWalking Tracer doc](https://www.envoyproxy.io/docs/envoy/v1.19.1/api-v3/config/trace/v3/skywalking.proto.html?highlight=skywalking) and [SkyWalking tracing sandbox](https://www.envoyproxy.io/docs/envoy/v1.19.1/start/sandboxes/skywalking_tracing?highlight=skywalking) for more details. diff --git a/docs/en/protocols/HTTP-API-Protocol.md b/docs/en/protocols/HTTP-API-Protocol.md deleted file mode 100644 index 8fd1a3411f..0000000000 --- a/docs/en/protocols/HTTP-API-Protocol.md +++ /dev/null @@ -1,186 +0,0 @@ -# HTTP API Protocol - -HTTP API Protocol defines the API data format, including API request and response data format. -They use the HTTP1.1 wrapper of the official [SkyWalking Trace Data Protocol v3](Trace-Data-Protocol-v3.md). Read it for more details. - -## Instance Management - -Detailed information about data format can be found in [Instance Management](https://github.com/apache/skywalking-data-collect-protocol/blob/master/management/Management.proto). - -- Report service instance properties - -> POST http://localhost:12800/v3/management/reportProperties - -Input: - -```json -{ - "service": "User Service Name", - "serviceInstance": "User Service Instance Name", - "properties": [ - { "key": "language", "value": "Lua" } - ] -} -``` - -Output JSON Array: - -```json -{} -``` - -- Service instance ping - -> POST http://localhost:12800/v3/management/keepAlive - -Input: - -```json -{ - "service": "User Service Name", - "serviceInstance": "User Service Instance Name" -} -``` - -OutPut: - -```json -{} -``` - -## Trace Report - -Detailed information about data format can be found in [Instance Management](https://github.com/apache/skywalking-data-collect-protocol/blob/master/language-agent/Tracing.proto). -There are two ways to report segment data: one segment per request or segment array in bulk mode. - -### POST http://localhost:12800/v3/segment - -Send a single segment object in JSON format. - -Input: - -```json -{ - "traceId": "a12ff60b-5807-463b-a1f8-fb1c8608219e", - "serviceInstance": "User_Service_Instance_Name", - "spans": [{ - "operationName": "/ingress", - "startTime": 1588664577013, - "endTime": 1588664577028, - "spanType": "Exit", - "spanId": 1, - "isError": false, - "parentSpanId": 0, - "componentId": 6000, - "peer": "upstream service", - "spanLayer": "Http" - }, { - "operationName": "/ingress", - "startTime": 1588664577013, - "tags": [{ - "key": "http.method", - "value": "GET" - }, { - "key": "http.params", - "value": "http://localhost/ingress"; - }], - "endTime": 1588664577028, - "spanType": "Entry", - "spanId": 0, - "parentSpanId": -1, - "isError": false, - "spanLayer": "Http", - "componentId": 6000 - }], - "service": "User_Service_Name", - "traceSegmentId": "a12ff60b-5807-463b-a1f8-fb1c8608219e" -} -``` - OutPut: - - ```json - -``` - -### POST http://localhost:12800/v3/segments - -Send a segment object list in JSON format. - -Input: - -```json -[{ - "traceId": "a12ff60b-5807-463b-a1f8-fb1c8608219e", - "serviceInstance": "User_Service_Instance_Name", - "spans": [{ - "operationName": "/ingress", - "startTime": 1588664577013, - "endTime": 1588664577028, - "spanType": "Exit", - "spanId": 1, - "isError": false, - "parentSpanId": 0, - "componentId": 6000, - "peer": "upstream service", - "spanLayer": "Http" - }, { - "operationName": "/ingress", - "startTime": 1588664577013, - "tags": [{ - "key": "http.method", - "value": "GET" - }, { - "key": "http.params", - "value": "http://localhost/ingress"; - }], - "endTime": 1588664577028, - "spanType": "Entry", - "spanId": 0, - "parentSpanId": -1, - "isError": false, - "spanLayer": "Http", - "componentId": 6000 - }], - "service": "User_Service_Name", - "traceSegmentId": "a12ff60b-5807-463b-a1f8-fb1c8608219e" -}, { - "traceId": "f956699e-5106-4ea3-95e5-da748c55bac1", - "serviceInstance": "User_Service_Instance_Name", - "spans": [{ - "operationName": "/ingress", - "startTime": 1588664577250, - "endTime": 1588664577250, - "spanType": "Exit", - "spanId": 1, - "isError": false, - "parentSpanId": 0, - "componentId": 6000, - "peer": "upstream service", - "spanLayer": "Http" - }, { - "operationName": "/ingress", - "startTime": 1588664577250, - "tags": [{ - "key": "http.method", - "value": "GET" - }, { - "key": "http.params", - "value": "http://localhost/ingress"; - }], - "endTime": 1588664577250, - "spanType": "Entry", - "spanId": 0, - "parentSpanId": -1, - "isError": false, - "spanLayer": "Http", - "componentId": 6000 - }], - "service": "User_Service_Name", - "traceSegmentId": "f956699e-5106-4ea3-95e5-da748c55bac1" -}] -``` - OutPut: - - ```json - -``` diff --git a/docs/en/protocols/JVM-Protocol.md b/docs/en/protocols/JVM-Protocol.md deleted file mode 100644 index 36214d4228..0000000000 --- a/docs/en/protocols/JVM-Protocol.md +++ /dev/null @@ -1,5 +0,0 @@ -# JVM Metrics Service -## Abstract -Uplink the JVM metrics, including PermSize, HeapSize, CPU, Memory, etc., every second. - -[gRPC service define](https://github.com/apache/skywalking-data-collect-protocol/blob/master/language-agent/JVMMetric.proto) diff --git a/docs/en/protocols/Log-Data-Protocol.md b/docs/en/protocols/Log-Data-Protocol.md deleted file mode 100644 index 5d909483a4..0000000000 --- a/docs/en/protocols/Log-Data-Protocol.md +++ /dev/null @@ -1,85 +0,0 @@ -# Log Data Protocol - -Report log data via protocol. - -## Native Proto Protocol - -Report `native-proto` format log via gRPC. - -[gRPC service define](https://github.com/apache/skywalking-data-collect-protocol/blob/master/logging/Logging.proto) - -## Native Kafka Protocol - -Report `native-json` format log via kafka. - -Json log record example: -```json -{ - "timestamp":1618161813371, - "service":"Your_ApplicationName", - "serviceInstance":"[email protected]", - "layer":"GENERAL", - "traceContext":{ - "traceId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001", - "spanId":"0", - "traceSegmentId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000" - }, - "tags":{ - "data":[ - { - "key":"level", - "value":"INFO" - }, - { - "key":"logger", - "value":"com.example.MyLogger" - } - ] - }, - "body":{ - "text":{ - "text":"log message" - } - } -} -``` - -## HTTP API - -Report `json` format logs via HTTP API, the endpoint is `http://<oap-address>:12800/v3/logs`. - -Json log record example: - -```json -[ - { - "timestamp": 1618161813371, - "service": "Your_ApplicationName", - "serviceInstance": "[email protected]", - "layer":"GENERAL", - "traceContext": { - "traceId": "ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001", - "spanId": "0", - "traceSegmentId": "ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000" - }, - "tags": { - "data": [ - { - "key": "level", - "value": "INFO" - }, - { - "key": "logger", - "value": "com.example.MyLogger" - } - ] - }, - "body": { - "text": { - "text": "log message" - } - } - } -] -``` - diff --git a/docs/en/protocols/README.md b/docs/en/protocols/README.md deleted file mode 100644 index 15a68b9241..0000000000 --- a/docs/en/protocols/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# Probe Protocols -Probe protocols describe and define how agents send collected metrics, logs, traces, and events, as well as set out the format of each entity. - -### Tracing -There are two types of protocols that help language agents work in distributed tracing. - -- **Cross Process Propagation Headers Protocol** and **Cross Process Correlation Headers Protocol** come in in-wire data format. Agent/SDK usually uses HTTP/MQ/HTTP2 headers to carry the data with the RPC request. The remote agent will receive this in the request handler, and bind the context with this specific request. - -[Cross Process Propagation Headers Protocol v3](Skywalking-Cross-Process-Propagation-Headers-Protocol-v3.md) has been the new protocol for in-wire context propagation since the version 8.0.0 release. - -[Cross Process Correlation Headers Protocol v1](Skywalking-Cross-Process-Correlation-Headers-Protocol-v1.md) is a new in-wire context propagation protocol which is additional and optional. -Please read SkyWalking language agents documentation to see whether it is supported. - -- **Trace Data Protocol** is an out-of-wire data format. Agent/SDK uses this to send traces to SkyWalking OAP server. - -[SkyWalking Trace Data Protocol v3.1](Trace-Data-Protocol-v3.md) defines the communication method and format between the agent and backend. - -### Logging -- **Log Data Protocol** is an out-of-wire data format. Agent/SDK and collector use this to send logs into SkyWalking OAP server. -[SkyWalking Log Data Protocol](Log-Data-Protocol.md) defines the communication method and format between the agent and backend. - -### Metrics - -SkyWalking has a native metrics format, and supports widely used metric formats, such as Prometheus, OpenCensus, OpenTelemetry, and Zabbix. - -The native metrics format definition could be found [here](https://github.com/apache/skywalking-data-collect-protocol/blob/master/language-agent/Meter.proto). -The agent meter plugin (e.g. [Java Meter Plugin](https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/java-plugin-development-guide/#meter-plugin)) uses the -native metric format to report metrics. - -OpenTelemetry collector, Telegraf agents, Zabbix agents could use their native protocol(e.g. OTLP) -and OAP server would convert metrics into native format and forward them to [MAL](../concepts-and-designs/mal.md) engine. - -To learn more about receiving 3rd party formats metrics, see [Meter receiver](../setup/backend/backend-meter.md) and [OpenTelemetry receiver](../setup/backend/opentelemetry-receiver.md). - -### Browser probe protocol - -The browser probe, such as [skywalking-client-js](https://github.com/apache/skywalking-client-js), could use this protocol to send data to the backend. This service is provided by gRPC. - -[SkyWalking Browser Protocol](Browser-Protocol.md) defines the communication method and format between `skywalking-client-js` and backend. - -### Events Report Protocol - -The protocol is used to report events to the backend. The [doc](../concepts-and-designs/event.md) introduces the definition of an event, and [the protocol repository](https://github.com/apache/skywalking-data-collect-protocol/blob/master/event) defines gRPC services and message formats of events. - -`JSON` format events can be reported via HTTP API. The endpoint is `http://<oap-address>:12800/v3/events`. -Example of a JSON event record: -```json -[ - { - "uuid": "f498b3c0-8bca-438d-a5b0-3701826ae21c", - "source": { - "service": "SERVICE-A", - "instance": "INSTANCE-1" - }, - "name": "Reboot", - "type": "Normal", - "message": "App reboot.", - "parameters": {}, - "startTime": 1628044330000, - "endTime": 1628044331000 - } -] -``` diff --git a/docs/en/setup/backend/log-analyzer.md b/docs/en/setup/backend/log-analyzer.md index 40edd87981..d61576f7a8 100644 --- a/docs/en/setup/backend/log-analyzer.md +++ b/docs/en/setup/backend/log-analyzer.md @@ -7,8 +7,8 @@ There are various ways to collect logs from applications. You can use [Filebeat](https://www.elastic.co/cn/beats/filebeat), [Fluentd](https://fluentd.org) and [FluentBit](http://fluentbit.io) to collect logs, and then transport the logs to SkyWalking OAP through Kafka or -HTTP protocol, with the formats [Kafka JSON](../../protocols/Log-Data-Protocol.md#native-kafka-protocol) -or [HTTP JSON array](../../protocols/Log-Data-Protocol.md#http-api). +HTTP protocol, with the formats [Kafka JSON](../../api/log-data-protocol.md#native-kafka-protocol) +or [HTTP JSON array](../../api/log-data-protocol.md#http-api). #### Filebeat Filebeat supports using Kafka to transport logs. Open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`. diff --git a/docs/en/setup/backend/mq.md b/docs/en/setup/backend/mq.md index 32dec4e16a..553835979b 100644 --- a/docs/en/setup/backend/mq.md +++ b/docs/en/setup/backend/mq.md @@ -5,7 +5,7 @@ blocking RPC and eventually improve user experience. But in this async way, the latency becomes significant. Since 8.9.0, SkyWalking leverages native tracing agent and [**Extension Header -Item** of SkyWalking Cross Process Propagation Headers Protocol v3](../../protocols/Skywalking-Cross-Process-Propagation-Headers-Protocol-v3.md#extension-header-item) +Item** of SkyWalking Cross Process Propagation Headers Protocol v3](../../api/x-process-propagation-headers-v3.md#extension-header-item) To provide performance monitoring for the Message Queue systems. In default, we provide `Message Queue Consuming Count` and `Message Queue Avg Consuming Latency` metrics for service and diff --git a/docs/menu.yml b/docs/menu.yml index 021eead392..f55b15976e 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -211,12 +211,36 @@ catalog: path: "/en/guides/how-to-release/" - name: "Compiling Guide" path: "/en/guides/How-to-build" - - name: "Protocols" + - name: "APIs" catalog: - - name: "Data Report(Probe/Agent) Protocol" - path: "/en/protocols/readme" - - name: "Query Protocol (GraphQL)" - path: "/en/protocols/query-protocol" + - name: "Telemetry APIs/Protocols" + catalog: + - name: "Tracing" + catalog: + - name: "Spans APIs" + path: "/en/api/trace-data-protocol-v3" + - name: "Tracing Context Propagation" + path: "/en/api/x-process-propagation-headers-v3" + - name: "Tracing Correlation Correlation" + path: "/en/api/x-process-correlation-headers-v1" + - name: "Metrics APIs" + catalog: + - name: "Meter APIs" + path: "/en/api/meter" + - name: "Browser performance APIs" + path: "en/api/browser-protocol" + - name: "JVM metrics APIs" + path: "/en/protocols/jvm-protocol" + - name: "Logging" + path: "/en/api/log-data-protocol" + - name: "Service Instance Properties APIs" + path: "/en/api/instance-properties" + - name: "Event" + path: "/en/api/event" + - name: "Query APIs" + catalog: + - name: "GraphQL APIs" + path: "/en/protocols/query-protocol" - name: "Security Notice" path: "/en/security/readme" - name: "Academy"
