[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-02 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r534211732



##
File path: docs/project_structue.md
##
@@ -0,0 +1,63 @@
+# Project Structure
+- configs: Satellite configs.
+- internal: Core, Api, and common utils.
+- internal/pkg: Sharing with Core and Plugins, such as api and utils.
+- internal/satellite: The core of Satellite.
+- plugins: Contains all plugins.
+- plugins/{type}: Contains the plugins of this {type}. Satellite has 6 plugin 
types, which are collector, queue, parser, filter, client, and forward.
+- plugins/{type}/define{type}: Contains the plugin define.

Review comment:
   could I merge it? According to chao's advice, the plugin was decoupled.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-02 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r534210310



##
File path: docs/project_structue.md
##
@@ -0,0 +1,63 @@
+# Project Structure
+- configs: Satellite configs.
+- internal: Core, Api, and common utils.
+- internal/pkg: Sharing with Core and Plugins, such as api and utils.
+- internal/satellite: The core of Satellite.
+- plugins: Contains all plugins.
+- plugins/{type}: Contains the plugins of this {type}. Satellite has 6 plugin 
types, which are collector, queue, parser, filter, client, and forward.
+- plugins/{type}/define{type}: Contains the plugin define.

Review comment:
   > What conflict?
   
   It's my mistake





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-02 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r534208738



##
File path: docs/project_structue.md
##
@@ -0,0 +1,63 @@
+# Project Structure
+- configs: Satellite configs.
+- internal: Core, Api, and common utils.
+- internal/pkg: Sharing with Core and Plugins, such as api and utils.
+- internal/satellite: The core of Satellite.
+- plugins: Contains all plugins.
+- plugins/{type}: Contains the plugins of this {type}. Satellite has 6 plugin 
types, which are collector, queue, parser, filter, client, and forward.
+- plugins/{type}/define{type}: Contains the plugin define.

Review comment:
   already fixed 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-02 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r534202928



##
File path: docs/project_structue.md
##
@@ -0,0 +1,63 @@
+# Project Structure
+- configs: Satellite configs.
+- internal: Core, Api, and common utils.
+- internal/pkg: Sharing with Core and Plugins, such as api and utils.
+- internal/satellite: The core of Satellite.
+- plugins: Contains all plugins.
+- plugins/{type}: Contains the plugins of this {type}. Satellite has 6 plugin 
types, which are collector, queue, parser, filter, client, and forward.
+- plugins/{type}/define{type}: Contains the plugin define.

Review comment:
   just to avoid package conflicts, let me try again.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-02 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r534199243



##
File path: docs/project_structue.md
##
@@ -0,0 +1,63 @@
+# Project Structure
+- configs: Satellite configs.
+- internal: Core, Api, and common utils.
+- internal/pkg: Sharing with Core and Plugins, such as api and utils.
+- internal/satellite: The core of Satellite.
+- plugins: Contains all plugins.
+- plugins/{type}: Contains the plugins of this {type}. Satellite has 6 plugin 
types, which are collector, queue, parser, filter, client, and forward.
+- plugins/{type}/define{type}: Contains the plugin define.

Review comment:
   move  API defines to plugins dir





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-01 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r533412645



##
File path: internal/pkg/api/event.go
##
@@ -0,0 +1,95 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import (
+   "fmt"
+   "time"
+)
+
+// The event type.
+const (
+   // Mapping to the type supported by SkyWalking OAP.
+   _ EventType = iota
+   MetricsEvent
+   ProfilingEvent
+   SegmentEvent
+   ManagementEvent
+   MeterEvent
+   LogEvent
+)
+
+type EventType int32
+
+// Event that implement this interface would be allowed to transmit in the 
Satellite.
+type Event interface {
+   // Name returns the event name.
+   Name() string
+
+   // Meta is a pair of key and value to record meta data, such as labels.
+   Meta() map[string]string
+
+   // Data returns the wrapped data.
+   Data() interface{}
+
+   // Time returns the event time.
+   Time() time.Time
+
+   // Type is to distinguish different events.
+   Type() EventType

Review comment:
   in OutputEventContext





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-01 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r533412315



##
File path: internal/pkg/api/forwarder.go
##
@@ -0,0 +1,42 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+//   Init() Initiating stage: Init plugin by config
+//||
+//\/
+//   Prepare()   Preparing stage: Prepare the Forwarder, such as get remote 
client.
+//||
+//\/
+//   Forward()  Running stage: Forward the batch events
+//||
+//\/
+//   Close()Closing stage: Close the Collector, such as close connection 
with SkyWalking javaagent.
+
+// Forwarder is a plugin interface, that defines new forwarders.
+type Forwarder interface {
+   Initializer
+   Preparer
+   Closer
+
+   // Forward the batch events to the external services, such as Kafka MQ 
and SkyWalking OAP cluster.
+   Forward(batch BatchEvents)
+
+   // ForwardType returns the supporting event type that could be 
forwarded.
+   ForwardType()

Review comment:
   sorry, adding it





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-12-01 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r533410861



##
File path: internal/satellite/event/struectured_event.go
##
@@ -0,0 +1,79 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package event
+
+import (
+   "time"
+
+   "github.com/apache/skywalking-satellite/internal/pkg/api"
+)
+
+// StructuredInputEventToBytesFunc serialize event to bytes.
+type StructuredInputEventToBytesFunc func(event *StructuredInputEvent) []byte
+
+// BytesToStructuredInputEventFunc deserialize bytes to event.
+type BytesToStructuredInputEventFunc func(bytes []byte) *StructuredInputEvent
+
+// StructuredEvent works when the data is a struct type.
+type StructuredEvent struct {

Review comment:
   for example segment.proto





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-30 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532560110



##
File path: internal/pkg/api/event.go
##
@@ -0,0 +1,103 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import (
+   "fmt"
+   "time"
+)
+
+// The event type.
+const (
+   // Mapping to the type supported by SkyWalking OAP.
+   _ EventType = iota
+   MetricsEvent
+   ProfilingEvent
+   SegmentEvent
+   ManagementEvent
+   MeterEvent
+   LogEvent
+)
+
+type EventType int32
+
+// Event that implement this interface would be allowed to transmit in the 
Satellite.
+type Event interface {
+   // Name returns the event name.
+   Name() string
+
+   // Meta is a pair of key and value to record meta data, such as labels.
+   Meta() map[string]string
+
+   // Data returns the wrapped data.
+   Data() interface{}
+
+   // Time returns the event time.
+   Time() time.Time
+
+   // Type is a identifier to distinguish different events.

Review comment:
   for distinguishing different events  to use different forwarders, such 
as log forwarder and metrics forwarder





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532381384



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following graph illustrates the relationship between different plugin 
interface in api package.
+//
+//
+//   GathererProcessor
+//   ---  
---
+//  |  ---   -   |   |  --- 
---  |
+//  | | Collector | ==> |  Queue   | |==>| |  Filter   | ==>  ...  ==> |  
Filter   | |
+//  | | (Parser)  | | Mem/File | |   |  --- 
---  |
+//  |  ---   --  |   |  || 
 ||   |
+//   |  \/ 
\/   |
+//   |  
---  |
+//   | | 
OutputEventContext| |
+//   |  
---  |
+//
---
+//||
+// ||

+// || ->| Sharing 
Client|
+// |||   

+// |||
+// \/|SegmentSender
+//---|  
-
+//   |   |   | |  ---   
---  |
+//   | D |   |-| |BatchBuffer| ==> | 
Forwarder | |
+//   | i |   | |  ---   
---  |
+//   | s |   |  
-
+//   | p |   |
+//   | a |=> |  ...
 ===> Kafka/OAP
+//   | t |   |
+//   | c |   | MeterSender
+//   | h |   | 
---
+//   | e |   -|  -   
---  |
+//   | r || | BatchBuffer | ==> | 
Forwarder | |
+//   |   ||  -   
---  |
+//---  
---
+//
+//
+// 1. The Collector plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to SerializationEvent that 
is supported
+//to be stored in Queue.
+// 3. The Queue plugin stores the SerializationEvent. However, the 
serialization depends on
+//the Queue implements. For example, the serialization is unnecessary when 
using a Memory
+//Queue.
+// 4. The Filter plugin would pull an event from the Queue and process the 
event to create
+//a new event. Next, the event is passed to the next filter to do the same 
things until
+//the whole filters are performed. The events labeled with RemoteEvent 
type would be
+//stored in the OutputEventContext. When the processing finished,
+//the OutputEventContext. After processing, the events in 
OutputEventContext would be
+//partitioned by the event type and sent to the different BatchBuffers, 
such as Segment
+//BatchBuffer, Jvm BatchBuffer, and Meter BatchBuffer.
+// 5. When each BatchBuffer reaches its maximum capacity, the 
OutputEventContexts would be

[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532225945



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following graph illustrates the relationship between different plugin 
interface in api package.
+//
+//
+//   GathererProcessor
+//   ---  
---
+//  |  ---   -   |   |  --- 
---  |
+//  | | Collector | ==> |  Queue   | |==>| |  Filter   | ==>  ...  ==> |  
Filter   | |
+//  | | (Parser)  | | Mem/File | |   |  --- 
---  |
+//  |  ---   --  |   |  || 
 ||   |
+//   |  \/ 
\/   |
+//   |  
---  |
+//   | | 
OutputEventContext| |
+//   |  
---  |
+//
---
+//||
+// ||

+// || ->| Sharing 
Client|
+// |||   

+// |||
+// \/|SegmentSender
+//---|  
-
+//   |   |   | |  ---   
---  |
+//   | D |   |-| |BatchBuffer| ==> | 
Forwarder | |
+//   | i |   | |  ---   
---  |
+//   | s |   |  
-
+//   | p |   |
+//   | a |=> |  ...
 ===> Kafka/OAP
+//   | t |   |
+//   | c |   | MeterSender
+//   | h |   | 
---
+//   | e |   -|  -   
---  |
+//   | r || | BatchBuffer | ==> | 
Forwarder | |
+//   |   ||  -   
---  |
+//---  
---
+//
+//
+// 1. The Collector plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to SerializationEvent that 
is supported
+//to be stored in Queue.
+// 3. The Queue plugin stores the SerializationEvent. However, the 
serialization depends on
+//the Queue implements. For example, the serialization is unnecessary when 
using a Memory
+//Queue.
+// 4. The Filter plugin would pull an event from the Queue and process the 
event to create
+//a new event. Next, the event is passed to the next filter to do the same 
things until
+//the whole filters are performed. The events labeled with RemoteEvent 
type would be

Review comment:
   add comments in plugin.go





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532225929



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following graph illustrates the relationship between different plugin 
interface in api package.
+//
+//
+//   GathererProcessor
+//   ---  
---
+//  |  ---   -   |   |  --- 
---  |
+//  | | Collector | ==> |  Queue   | |==>| |  Filter   | ==>  ...  ==> |  
Filter   | |
+//  | | (Parser)  | | Mem/File | |   |  --- 
---  |
+//  |  ---   --  |   |  || 
 ||   |
+//   |  \/ 
\/   |
+//   |  
---  |
+//   | | 
OutputEventContext| |
+//   |  
---  |
+//
---
+//||
+// ||

+// || ->| Sharing 
Client|
+// |||   

+// |||
+// \/|SegmentSender
+//---|  
-
+//   |   |   | |  ---   
---  |
+//   | D |   |-| |BatchBuffer| ==> | 
Forwarder | |
+//   | i |   | |  ---   
---  |
+//   | s |   |  
-
+//   | p |   |
+//   | a |=> |  ...
 ===> Kafka/OAP
+//   | t |   |
+//   | c |   | MeterSender
+//   | h |   | 
---
+//   | e |   -|  -   
---  |
+//   | r || | BatchBuffer | ==> | 
Forwarder | |
+//   |   ||  -   
---  |
+//---  
---
+//
+//
+// 1. The Collector plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to SerializationEvent that 
is supported
+//to be stored in Queue.
+// 3. The Queue plugin stores the SerializationEvent. However, the 
serialization depends on
+//the Queue implements. For example, the serialization is unnecessary when 
using a Memory
+//Queue.
+// 4. The Filter plugin would pull an event from the Queue and process the 
event to create
+//a new event. Next, the event is passed to the next filter to do the same 
things until
+//the whole filters are performed. The events labeled with RemoteEvent 
type would be
+//stored in the OutputEventContext. When the processing finished,
+//the OutputEventContext. After processing, the events in 
OutputEventContext would be
+//partitioned by the event type and sent to the different BatchBuffers, 
such as Segment
+//BatchBuffer, Jvm BatchBuffer, and Meter BatchBuffer.
+// 5. When each BatchBuffer reaches its maximum capacity, the 
OutputEventContexts would be

[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r53379



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following graph illustrates the relationship between different plugin 
interface in api package.
+//
+//
+//   GathererProcessor
+//   ---  
---
+//  |  ---   -   |   |  --- 
---  |
+//  | | Collector | ==> |  Queue   | |==>| |  Filter   | ==>  ...  ==> |  
Filter   | |
+//  | | (Parser)  | | Mem/File | |   |  --- 
---  |
+//  |  ---   --  |   |  || 
 ||   |
+//   |  \/ 
\/   |
+//   |  
---  |
+//   | | 
OutputEventContext| |
+//   |  
---  |
+//
---
+//||
+// ||

+// || ->| Sharing 
Client|
+// |||   

+// |||
+// \/|SegmentSender
+//---|  
-
+//   |   |   | |  ---   
---  |
+//   | D |   |-| |BatchBuffer| ==> | 
Forwarder | |
+//   | i |   | |  ---   
---  |
+//   | s |   |  
-
+//   | p |   |
+//   | a |=> |  ...
 ===> Kafka/OAP
+//   | t |   |
+//   | c |   | MeterSender
+//   | h |   | 
---
+//   | e |   -|  -   
---  |
+//   | r || | BatchBuffer | ==> | 
Forwarder | |
+//   |   ||  -   
---  |
+//---  
---
+//
+//
+// 1. The Collector plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to SerializationEvent that 
is supported
+//to be stored in Queue.
+// 3. The Queue plugin stores the SerializationEvent. However, the 
serialization depends on
+//the Queue implements. For example, the serialization is unnecessary when 
using a Memory
+//Queue.
+// 4. The Filter plugin would pull an event from the Queue and process the 
event to create
+//a new event. Next, the event is passed to the next filter to do the same 
things until
+//the whole filters are performed. The events labeled with RemoteEvent 
type would be
+//stored in the OutputEventContext. When the processing finished,
+//the OutputEventContext. After processing, the events in 
OutputEventContext would be
+//partitioned by the event type and sent to the different BatchBuffers, 
such as Segment
+//BatchBuffer, Jvm BatchBuffer, and Meter BatchBuffer.
+// 5. When each BatchBuffer reaches its maximum capacity, the 
OutputEventContexts would be

[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532221356



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following graph illustrates the relationship between different plugin 
interface in api package.
+//
+//
+//   GathererProcessor
+//   ---  
---
+//  |  ---   -   |   |  --- 
---  |
+//  | | Collector | ==> |  Queue   | |==>| |  Filter   | ==>  ...  ==> |  
Filter   | |
+//  | | (Parser)  | | Mem/File | |   |  --- 
---  |
+//  |  ---   --  |   |  || 
 ||   |
+//   |  \/ 
\/   |
+//   |  
---  |
+//   | | 
OutputEventContext| |
+//   |  
---  |
+//
---
+//||
+// ||

+// || ->| Sharing 
Client|
+// |||   

+// |||
+// \/|SegmentSender
+//---|  
-
+//   |   |   | |  ---   
---  |
+//   | D |   |-| |BatchBuffer| ==> | 
Forwarder | |
+//   | i |   | |  ---   
---  |
+//   | s |   |  
-
+//   | p |   |
+//   | a |=> |  ...
 ===> Kafka/OAP
+//   | t |   |
+//   | c |   | MeterSender
+//   | h |   | 
---
+//   | e |   -|  -   
---  |
+//   | r || | BatchBuffer | ==> | 
Forwarder | |
+//   |   ||  -   
---  |
+//---  
---
+//
+//
+// 1. The Collector plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to SerializationEvent that 
is supported
+//to be stored in Queue.
+// 3. The Queue plugin stores the SerializationEvent. However, the 
serialization depends on
+//the Queue implements. For example, the serialization is unnecessary when 
using a Memory
+//Queue.
+// 4. The Filter plugin would pull an event from the Queue and process the 
event to create
+//a new event. Next, the event is passed to the next filter to do the same 
things until
+//the whole filters are performed. The events labeled with RemoteEvent 
type would be
+//stored in the OutputEventContext. When the processing finished,
+//the OutputEventContext. After processing, the events in 
OutputEventContext would be
+//partitioned by the event type and sent to the different BatchBuffers, 
such as Segment
+//BatchBuffer, Jvm BatchBuffer, and Meter BatchBuffer.
+// 5. When each BatchBuffer reaches its maximum capacity, the 
OutputEventContexts would be

[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532221267



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following graph illustrates the relationship between different plugin 
interface in api package.
+//
+//
+//   GathererProcessor
+//   ---  
---
+//  |  ---   -   |   |  --- 
---  |
+//  | | Collector | ==> |  Queue   | |==>| |  Filter   | ==>  ...  ==> |  
Filter   | |
+//  | | (Parser)  | | Mem/File | |   |  --- 
---  |
+//  |  ---   --  |   |  || 
 ||   |
+//   |  \/ 
\/   |
+//   |  
---  |
+//   | | 
OutputEventContext| |
+//   |  
---  |
+//
---
+//||
+// ||

+// || ->| Sharing 
Client|
+// |||   

+// |||
+// \/|SegmentSender
+//---|  
-
+//   |   |   | |  ---   
---  |
+//   | D |   |-| |BatchBuffer| ==> | 
Forwarder | |
+//   | i |   | |  ---   
---  |
+//   | s |   |  
-
+//   | p |   |
+//   | a |=> |  ...
 ===> Kafka/OAP
+//   | t |   |
+//   | c |   | MeterSender
+//   | h |   | 
---
+//   | e |   -|  -   
---  |
+//   | r || | BatchBuffer | ==> | 
Forwarder | |
+//   |   ||  -   
---  |
+//---  
---
+//
+//
+// 1. The Collector plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to SerializationEvent that 
is supported
+//to be stored in Queue.
+// 3. The Queue plugin stores the SerializationEvent. However, the 
serialization depends on
+//the Queue implements. For example, the serialization is unnecessary when 
using a Memory
+//Queue.
+// 4. The Filter plugin would pull an event from the Queue and process the 
event to create
+//a new event. Next, the event is passed to the next filter to do the same 
things until
+//the whole filters are performed. The events labeled with RemoteEvent 
type would be

Review comment:
   Do u means the following things? 
   
![image](https://user-images.githubusercontent.com/31562192/100545366-77b89200-3296-11eb-8c76-eb60dd3a20bf.png)
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.


[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532220022



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following graph illustrates the relationship between different plugin 
interface in api package.
+//
+//
+//   GathererProcessor
+//   ---  
---
+//  |  ---   -   |   |  --- 
---  |
+//  | | Collector | ==> |  Queue   | |==>| |  Filter   | ==>  ...  ==> |  
Filter   | |
+//  | | (Parser)  | | Mem/File | |   |  --- 
---  |
+//  |  ---   --  |   |  || 
 ||   |
+//   |  \/ 
\/   |
+//   |  
---  |
+//   | | 
OutputEventContext| |
+//   |  
---  |
+//
---
+//||
+// ||

+// || ->| Sharing 
Client|
+// |||   

+// |||
+// \/|SegmentSender
+//---|  
-
+//   |   |   | |  ---   
---  |
+//   | D |   |-| |BatchBuffer| ==> | 
Forwarder | |
+//   | i |   | |  ---   
---  |
+//   | s |   |  
-
+//   | p |   |
+//   | a |=> |  ...
 ===> Kafka/OAP
+//   | t |   |
+//   | c |   | MeterSender
+//   | h |   | 
---
+//   | e |   -|  -   
---  |
+//   | r || | BatchBuffer | ==> | 
Forwarder | |
+//   |   ||  -   
---  |
+//---  
---
+//
+//
+// 1. The Collector plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to SerializationEvent that 
is supported
+//to be stored in Queue.
+// 3. The Queue plugin stores the SerializationEvent. However, the 
serialization depends on
+//the Queue implements. For example, the serialization is unnecessary when 
using a Memory
+//Queue.
+// 4. The Filter plugin would pull an event from the Queue and process the 
event to create

Review comment:
   right, would polish the comments





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-29 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532207523



##
File path: internal/satellite/registry/registry.go
##
@@ -0,0 +1,168 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package registry
+
+import (
+   "fmt"
+
+   "github.com/apache/skywalking-satellite/internal/pkg/api"
+)
+
+// The creator reg.
+type pluginRegistry struct {
+   collectorCreatorRegistry map[string]CollectorCreator
+   queueCreatorRegistry map[string]QueueCreator
+   filterCreatorRegistrymap[string]FilterCreator
+   forwarderCreatorRegistry map[string]ForwarderCreator
+   parserCreatorRegistrymap[string]ParserCreator
+   clientCreatorRegistrymap[string]ClientCreator
+}
+
+// ClientCreator creates a Client according to the config.
+type ClientCreator func(config map[string]interface{}) (api.Collector, error)
+
+// CollectorCreator creates a Collector according to the config.
+type CollectorCreator func(config map[string]interface{}) (api.Collector, 
error)
+
+// QueueCreator creates a Queue according to the config.
+type QueueCreator func(config map[string]interface{}) (api.Queue, error)
+
+// FilterCreator creates a Filter according to the config.
+type FilterCreator func(config map[string]interface{}) (api.Filter, error)
+
+// ForwarderCreator creates a forwarder according to the config.
+type ForwarderCreator func(config map[string]interface{}) (api.Forwarder, 
error)
+
+// ParserCreator creates a parser according to the config.
+type ParserCreator func(config map[string]interface{}) (api.Parser, error)
+
+var reg *pluginRegistry
+
+// RegisterClient registers the clientType as ClientCreator.
+func RegisterClient(clientType string, creator ClientCreator) {
+   fmt.Printf("register client creator success : %s", clientType)

Review comment:
   the creator is a func to create a new instance, its name is stored in a 
map. I think that is enough to use. In the future code, I would implement 
plugins in ./plugins dir.  The plugin type would be as the next level dirs, 
such as collector, client, or queue. And the 3rd level would be the plugin 
name. Maybe I would like to implement a method to read the parent dir as the 
plugin name.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-28 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532053497



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.
+//
+//
+// Processors
+//   -
+//  ----- ---
+// | Gatherer | ==> |  Queue   | ==> | Processor | ==>  ...  ==> | Processor |
+// | (Parser) | | Mem/File |  --- ---
+//  --   -||  ||
+//\/ \/
+//---
+//   | OutputEventContext|
+//---
+//||
+//\/
+//---   -
+//   | BatchOutputEvents | <== | BatchBuffer |
+//---   -
+// ||
+// \/
+//---
+//   | Forwarder | ==> Kakfa/OAP
+//---
+// 1. The Gatherer plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to InputEvent.
+//If the event needs output, please tag it by the IsOutput
+//method.
+// 3. The Queue plugin would store the InputEvent. But different

Review comment:
   Maybe SerializeEvent more fit the Queue?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-28 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532052890



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.
+//
+//
+// Processors
+//   -
+//  ----- ---
+// | Gatherer | ==> |  Queue   | ==> | Processor | ==>  ...  ==> | Processor |
+// | (Parser) | | Mem/File |  --- ---
+//  --   -||  ||
+//\/ \/
+//---
+//   | OutputEventContext|
+//---
+//||
+//\/
+//---   -
+//   | BatchOutputEvents | <== | BatchBuffer |
+//---   -
+// ||
+// \/
+//---
+//   | Forwarder | ==> Kakfa/OAP
+//---
+// 1. The Gatherer plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to InputEvent.
+//If the event needs output, please tag it by the IsOutput

Review comment:
   That's only a switch case to support more cases.  Do u have other 
suggestions?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-28 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532040371



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.
+//
+//
+// Processors
+//   -
+//  ----- ---
+// | Gatherer | ==> |  Queue   | ==> | Processor | ==>  ...  ==> | Processor |
+// | (Parser) | | Mem/File |  --- ---
+//  --   -||  ||
+//\/ \/
+//---
+//   | OutputEventContext|
+//---
+//||
+//\/
+//---   -
+//   | BatchOutputEvents | <== | BatchBuffer |
+//---   -
+// ||
+// \/
+//---
+//   | Forwarder | ==> Kakfa/OAP
+//---
+// 1. The Gatherer plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to InputEvent.
+//If the event needs output, please tag it by the IsOutput
+//method.
+// 3. The Queue plugin would store the InputEvent. But different
+//Queue would use different ways to store data, such as store
+//bytes by serialization or keep original.
+// 4. The Processor plugin would pull the event from the Queue and
+//process the event to create a new event. Next, the event is
+//passed to the next processor to do the same things until the
+//whole processors are performed. Similar to above, if any
+//events need output, please mark. The events would be stored
+//in the OutputEventContext. When the processing is finished,
+//the OutputEventContext would be passed to the BatchBuffer.
+// 5. When BatchBuffer is full, the OutputEventContexts would be
+//partitioned by event name and convert to BatchOutputEvents.
+// 6. The Follower would be ordered to send each partition in
+//BatchOutputEvents in different ways, such as different gRPC
+//endpoints.
+
+// ComponentPlugin is an interface to initialize the specific plugin.
+type ComponentPlugin interface {
+   io.Closer
+
+   // Init initialize the specific plugin and would return error when the 
configuration is error.
+   Init() error
+
+   // Run triggers the specific plugin to work.
+   Run()

Review comment:
   Init method means to create.  The run method means to start, such as 
build connection. The process or publish method is core processing when running.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-28 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532039980



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.
+//
+//
+// Processors
+//   -
+//  ----- ---
+// | Gatherer | ==> |  Queue   | ==> | Processor | ==>  ...  ==> | Processor |
+// | (Parser) | | Mem/File |  --- ---
+//  --   -||  ||
+//\/ \/
+//---
+//   | OutputEventContext|
+//---
+//||
+//\/
+//---   -
+//   | BatchOutputEvents | <== | BatchBuffer |
+//---   -
+// ||
+// \/
+//---
+//   | Forwarder | ==> Kakfa/OAP
+//---
+// 1. The Gatherer plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to InputEvent.
+//If the event needs output, please tag it by the IsOutput

Review comment:
   When we support sampling, no output event type needs to be supported to 
keep the filter chain going. The Output options should be decided by the 
Processor. That means the input event should always false. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-28 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532039980



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.
+//
+//
+// Processors
+//   -
+//  ----- ---
+// | Gatherer | ==> |  Queue   | ==> | Processor | ==>  ...  ==> | Processor |
+// | (Parser) | | Mem/File |  --- ---
+//  --   -||  ||
+//\/ \/
+//---
+//   | OutputEventContext|
+//---
+//||
+//\/
+//---   -
+//   | BatchOutputEvents | <== | BatchBuffer |
+//---   -
+// ||
+// \/
+//---
+//   | Forwarder | ==> Kakfa/OAP
+//---
+// 1. The Gatherer plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to InputEvent.
+//If the event needs output, please tag it by the IsOutput

Review comment:
   When we support sampling, no output needs to be supported. The Output 
options should be decided by the Processor. That means the input event should 
always false. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-28 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r532034993



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.
+//
+//
+// Processors
+//   -
+//  ----- ---
+// | Gatherer | ==> |  Queue   | ==> | Processor | ==>  ...  ==> | Processor |
+// | (Parser) | | Mem/File |  --- ---
+//  --   -||  ||
+//\/ \/
+//---
+//   | OutputEventContext|
+//---
+//||
+//\/
+//---   -
+//   | BatchOutputEvents | <== | BatchBuffer |
+//---   -
+// ||
+// \/
+//---
+//   | Forwarder | ==> Kakfa/OAP
+//---
+// 1. The Gatherer plugin would fetch or receive the input data.
+// 2. The Parser plugin would parse the input data to InputEvent.
+//If the event needs output, please tag it by the IsOutput
+//method.
+// 3. The Queue plugin would store the InputEvent. But different
+//Queue would use different ways to store data, such as store
+//bytes by serialization or keep original.
+// 4. The Processor plugin would pull the event from the Queue and

Review comment:
   no problem, I'm going to follow the document naming





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-27 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r531692867



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.

Review comment:
   Here is the detailed thinking on the plugins interface. @wu-sheng 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [skywalking-satellite] EvanLjp commented on a change in pull request #5: Add API && Plugin framework registry

2020-11-27 Thread GitBox


EvanLjp commented on a change in pull request #5:
URL: https://github.com/apache/skywalking-satellite/pull/5#discussion_r531692867



##
File path: internal/pkg/api/plugin.go
##
@@ -0,0 +1,74 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package api
+
+import "io"
+
+// The following comments is to illustrate the relationship between different 
plugin interface in api package.

Review comment:
   Here is the detailed thinking on the plugins interface. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org