Great, this is a super important feature. Let's do this. Sheng Wu 吴晟 Twitter, wusheng1108
Ming Wen <wenm...@apache.org> 于2021年4月10日周六 上午8:42写道: > APISIX can supoort gRPC sender in one week, I discussed with yuan sheng > wang yesterday. > So we can use gRPC to send ALS to skywalking. > > Sheng Wu <wu.sheng.841...@gmail.com>于2021年4月9日 周五下午7:44写道: > > > Let's move this to review on GitHub PR, which should be more friendly. > > > > Sheng Wu 吴晟 > > Twitter, wusheng1108 > > > > > > wei jin <k...@apache.org> 于2021年4月9日周五 下午7:05写道: > > > > > hello, community, > > > > > > After some discussion, I made a summary and listed some function points > > > that need to be collaborated. The content is as follows: > > > > > > - Protocol > > > Firstly, use http protocol to send access log to skywalking, and in the > > > future use GRPC protocol. > > > > > > - Structure > > > AccessLogProto (ALP) [1] > > > > > > - Community Cooperation > > > > > > Apache APISIX > > > 1. Support the collection of fields required by access log > > > 2. Extend plugin, use http protocol to send to the target address > > > > > > Apache Skywalking > > > 1. Implement http receiver, support AccessLogProto (ALP) format > analysis, > > > and write to OAP > > > > > > Welcome to add and discuss, thanks. > > > > > > [1] Data structure definition for reference > > > ```go > > > message AccessLogEntry { > > > AccessLogCommon common = 1; > > > HTTPRequestProperties request = 2; > > > HTTPResponseProperties response = 3; > > > } > > > ``` > > > > > > ```go > > > message AccessLogCommon { > > > double sample_rate = 1; > > > string downstream_remote_address = 2; > > > string downstream_local_address = 3; > > > string upstream_remote_address = 4; > > > string upstream_local_address = 5; > > > google.protobuf.Timestamp start_time = 6; > > > // Interval between the first downstream byte received and the first > > > upstream byte sent > > > google.protobuf.Duration time_to_first_upstream_tx_byte = 7; > > > string route_name = 8; > > > HTTPRequestProperties request = 9; > > > HTTPResponseProperties response = 10; > > > } > > > ``` > > > > > > > > > ```go > > > message HTTPRequestProperties { > > > string method = 1; > > > string scheme = 2; > > > google.protobuf.UInt32Value port = 3; > > > string path = 4; > > > string user_agent = 5; > > > string forwarded_for = 6; > > > tring request_id = 7; > > > uint64 request_headers_bytes = 8; > > > uint64 request_body_bytes = 9; > > > map<string, string> request_headers = 10; > > > } > > > ``` > > > > > > ```go > > > message HTTPResponseProperties { > > > uint64 response_code = 1; > > > uint64 response_headers_bytes = 2; > > > uint64 response_body_bytes = 3; > > > map<string, string> response_headers = 4; > > > } > > > ``` > > > > > > Sheng Wu <wu.sheng.841...@gmail.com> 于2021年4月6日周二 下午4:45写道: > > > > > > > This seems a very abstract question. > > > > What kind of access log do you have? Is there a design or format > > document > > > > to describe that? > > > > > > > > Sheng Wu 吴晟 > > > > Twitter, wusheng1108 > > > > > > > > > > > > Ming Wen <wenm...@apache.org> 于2021年4月6日周二 下午4:17写道: > > > > > > > > > Hi, sheng wu, > > > > > How can Apache APISIX send ALS to skywalking? > > > > > Are we implementing it in the existing skywalking plugin? Or need > to > > > add > > > > a > > > > > new plug-in? > > > > > > > > > > Thanks, > > > > > Ming Wen, Apache APISIX PMC Chair > > > > > Twitter: _WenMing > > > > > > > > > > > > > > > Sheng Wu <wu.sheng.841...@gmail.com> 于2021年3月23日周二 下午12:07写道: > > > > > > > > > > > Ming Wen <wenm...@apache.org> 于2021年3月23日周二 上午11:25写道: > > > > > > > > > > > > > > The biggest advantage of ALS, is replacing the physical local > > > > access > > > > > > log > > > > > > > > file. So you have metrics, topology(dependency), and logs in > > one > > > > > > package > > > > > > > of > > > > > > > > data. > > > > > > > > > > > > > > If we also record the metrics and tracing data with the > physical > > > > local > > > > > > > access log, can we achieve the same effect? > > > > > > > > > > > > > > > > > > > Technically, it is practicable. Once we add file scanning > > capability > > > in > > > > > the > > > > > > satellite, or use 3rd party tool, and forward to the OAP. > > > > > > The question is, do we really need the physical log file, as > > > basically, > > > > > it > > > > > > costs unnecessary resources. > > > > > > > > > > > > > > > > > > Sheng Wu 吴晟 > > > > > > Twitter, wusheng1108 > > > > > > > > > > > > > > > > > > > > > > > > > > > If you want to discuss most ALS in APISIX, I am glad to help > > with > > > > the > > > > > > > > design and integration solution. > > > > > > > > > > > > > > yes, I want to discuss more details. > > > > > > > > > > > > > > Thanks, > > > > > > > Ming Wen, Apache APISIX PMC Chair > > > > > > > Twitter: _WenMing > > > > > > > > > > > > > > > > > > > > > wei jin <k...@apache.org> 于2021年3月22日周一 下午3:58写道: > > > > > > > > > > > > > > > Zexuan Luo <spacewan...@apache.org> 于2021年3月21日周日 下午2:18写道: > > > > > > > > > > > > > > > > > I am not familiar with > > > > > > https://github.com/apache/skywalking-satellite. > > > > > > > > > Can we use this project to convert plain HTTP to GRPC? > > > > > > > > > > > > > > > > > > > > > > > > Skywalking satellite is usually deployed as a sidecar. It can > > > > convert > > > > > > > > APISIX gateway metrics to other data formats, do some > > > > preprocessing, > > > > > > and > > > > > > > > even change the data collection method from pull to push. > > > > > > > > > > > > > > > > However, the original prometheus metrics are still generated > by > > > > > APISIX, > > > > > > > > which is not suitable for URI-level statistics. > > > > > > > > > > > > > > > > wei jin <k...@apache.org> 于2021年3月22日周一 下午3:57写道: > > > > > > > > > > > > > > > > > Zexuan Luo <spacewan...@apache.org> 于2021年3月21日周日 > 下午2:18写道: > > > > > > > > > > > > > > > > > > > I am not familiar with > > > > > > > https://github.com/apache/skywalking-satellite. > > > > > > > > > > Can we use this project to convert plain HTTP to GRPC? > > > > > > > > > > > > > > > > > > > > > > > > > > > Skywalking satellite is usually deployed as a sidecar. It > can > > > > > convert > > > > > > > > > APISIX gateway metrics to other data formats, do some > > > > > preprocessing, > > > > > > > and > > > > > > > > > even change the data collection method from pull to push. > > > > > > > > > > > > > > > > > > However, the original prometheus metrics are still > generated > > by > > > > > > APISIX, > > > > > > > > > which is not suitable for URI-level statistics. > > > > > > > > > > > > > > > > > > > > > > > > > > > YuanSheng Wang <membp...@apache.org> 于2021年3月21日周日 > > 下午10:08写道: > > > > > > > > > > > > > > > > > >> For APISIX, gRPC calls need to be supported. > > > > > > > > >> > > > > > > > > >> Either subrequest or pure Lua mode is ok, this feature is > > also > > > > > very > > > > > > > > useful > > > > > > > > >> in other cases. > > > > > > > > >> > > > > > > > > >> We can use `subrequest` to implement the first version, > this > > > way > > > > > > seems > > > > > > > > >> simpler. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On Sun, Mar 21, 2021 at 2:34 PM Zexuan Luo < > > > > > spacewan...@apache.org> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >> > Another solution is writing a Lua GRPC library with > > > > > > > > >> > https://github.com/tokers/lua-resty-http2 > > > > > > > > >> > https://github.com/starwing/lua-protobuf > > > > > > > > >> > > > > > > > > > >> > Zexuan Luo <spacewan...@apache.org> 于2021年3月21日周日 > > 下午2:18写道: > > > > > > > > >> > > > > > > > > > > >> > > I am not familiar with > > > > > > > > https://github.com/apache/skywalking-satellite > > > > > > > > >> . > > > > > > > > >> > > Can we use this project to convert plain HTTP to GRPC? > > > > > > > > >> > > > > > > > > > > >> > > Sheng Wu <wu.sheng.841...@gmail.com> 于2021年3月21日周日 > > > > 下午1:59写道: > > > > > > > > >> > > > > > > > > > > > >> > > > Oh, you mean that. > > > > > > > > >> > > > SkyWalking natively provides HTTP APIs for nearly > all > > > > > > protocols, > > > > > > > > >> even > > > > > > > > >> > if > > > > > > > > >> > > > not, we are happy to add. There is no block between > > > APISIX > > > > > and > > > > > > > > >> > SkyWalking. > > > > > > > > >> > > > The point to discuss the gRPC service here is about > > the > > > > > > > > performance, > > > > > > > > >> > > > including proto's no field name in transmission, > enum > > as > > > > > int, > > > > > > > > small > > > > > > > > >> > int as > > > > > > > > >> > > > small, and especially streaming. > > > > > > > > >> > > > If you want to implement ALS, then this is super > > > important > > > > > to > > > > > > > > >> provide > > > > > > > > >> > > > better performance. > > > > > > > > >> > > > > > > > > > > > >> > > > Sheng Wu 吴晟 > > > > > > > > >> > > > Twitter, wusheng1108 > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > Chao Zhang <zchao1...@gmail.com> 于2021年3月21日周日 > > > 下午1:55写道: > > > > > > > > >> > > > > > > > > > > > >> > > > > It’s not so easy to use gRPC in APISIX as the lack > > of > > > > > > > ecosystem, > > > > > > > > >> but > > > > > > > > >> > with > > > > > > > > >> > > > > the help of gRPC gateway ( > > > > > > > > >> > https://github.com/grpc-ecosystem/grpc-gateway). > > > > > > > > >> > > > > We can use the restful APIs to talk with gRPC > > > endpoints. > > > > > > Just > > > > > > > > like > > > > > > > > >> > the way > > > > > > > > >> > > > > APISIX communicates with ETCD V3 APIs. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Chao Zhang > > > > > > > > >> > > > > https://github.com/tokers > > > > > > > > >> > > > > > > > > > > > > >> > > > > On March 21, 2021 at 1:53:41 PM, Sheng Wu ( > > > > > > > > >> wu.sheng.841...@gmail.com > > > > > > > > >> > ) > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > > Chao Zhang <zchao1...@gmail.com> 于2021年3月21日周日 > > > > 下午1:46写道: > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Does the gRPC service enables grpc gateway? If > so, > > > > it’s > > > > > > easy > > > > > > > > for > > > > > > > > >> > APISIX > > > > > > > > >> > > > > to > > > > > > > > >> > > > > > integrate it? > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > What is the relationship with grpc gateway? > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Zexuan Luo <spacewan...@apache.org>于2021年3月21日 > > > > > 周日13:35写道: > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Another solution is wrapping the C++ > > > implementation > > > > of > > > > > > the > > > > > > > > >> GPRC > > > > > > > > >> > > > > > > client. But it requires you to prebuild the > > proto > > > > file > > > > > > > into > > > > > > > > >> *.c > > > > > > > > >> > source > > > > > > > > >> > > > > > > and can only be distributed as part of > > > > > apisix-openresty. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Zexuan Luo <spacewan...@apache.org> > > 于2021年3月21日周日 > > > > > > > 下午1:25写道: > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > AFAIK, there is not library support making > > GPRC > > > > > > > directly. > > > > > > > > >> > > > > > > > One solution is subrequest + http2grpc, but > > this > > > > way > > > > > > > > doesn't > > > > > > > > >> > support > > > > > > > > >> > > > > > > > stream mode. So it is not a good solution. > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > YuanSheng Wang <membp...@apache.org> > > > > 于2021年3月20日周六 > > > > > > > > >> 下午10:53写道: > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > LGTM > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > The biggest advantage of ALS, is > replacing > > > the > > > > > > > > physical > > > > > > > > >> > local > > > > > > > > >> > > > > > access > > > > > > > > >> > > > > > > log > > > > > > > > >> > > > > > > > > file. > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > The APISIX has a plugin `http-logger`, I > > think > > > > > > APISIX > > > > > > > is > > > > > > > > >> > easy to > > > > > > > > >> > > > > > report > > > > > > > > >> > > > > > > > > those data to SkyWalking. > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > And I think we can try to use `gRPC` to > > report > > > > > data. > > > > > > > > What > > > > > > > > >> do > > > > > > > > >> > > > > > > > > you think? @ming @spacewander > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > On Fri, Mar 19, 2021 at 11:43 PM Ming Wen > < > > > > > > > > >> > wenm...@apache.org> > > > > > > > > >> > > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > Cool, I think we can try ALS > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > Sheng Wu <wu.sheng.841...@gmail.com > > > > >于2021年3月19日 > > > > > > > > >> > 周五下午11:21写道: > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > We have run the Envoy ALS performance > > > test a > > > > > > long > > > > > > > > time > > > > > > > > >> > ago(2 > > > > > > > > >> > > > > > > years, I > > > > > > > > >> > > > > > > > > > > think). Sadly, no public blog. > > > > > > > > >> > > > > > > > > > > It could easily support 10k-20k/s > > traffic > > > > > with a > > > > > > > > very > > > > > > > > >> > low load > > > > > > > > >> > > > > to > > > > > > > > >> > > > > > > the > > > > > > > > >> > > > > > > > > > > proxy. But, it is based on gRPC, so > > > > basically, > > > > > > it > > > > > > > > >> > doesn't send > > > > > > > > >> > > > > > many > > > > > > > > >> > > > > > > > > > things > > > > > > > > >> > > > > > > > > > > out, mostly only flags and latency > > value. > > > > > > > > >> > > > > > > > > > > APISIX is using HTTP1.1 mostly, so, > this > > > > could > > > > > > be > > > > > > > > >> > different. > > > > > > > > >> > > > > > > > > > > Also, at the same time, besides the > ALS, > > > we > > > > > > have a > > > > > > > > >> > chance to > > > > > > > > >> > > > > > > provide a > > > > > > > > >> > > > > > > > > > push > > > > > > > > >> > > > > > > > > > > mode LUA SDK in skywalking meter > format. > > > It > > > > > > could > > > > > > > > >> provide > > > > > > > > >> > > > > > URI-level > > > > > > > > >> > > > > > > > > > metrics > > > > > > > > >> > > > > > > > > > > at reasonable resource cost, > especially > > it > > > > > works > > > > > > > > with > > > > > > > > >> > > > > SkyWalking > > > > > > > > >> > > > > > > > > > satellite, > > > > > > > > >> > > > > > > > > > > which could leverage local disk to > cache > > > the > > > > > > data. > > > > > > > > >> > > > > > > > > > > The biggest advantage of ALS, is > > replacing > > > > the > > > > > > > > >> physical > > > > > > > > >> > local > > > > > > > > >> > > > > > > access log > > > > > > > > >> > > > > > > > > > > file. So you have metrics, > > > > > topology(dependency), > > > > > > > and > > > > > > > > >> > logs in > > > > > > > > >> > > > > one > > > > > > > > >> > > > > > > package > > > > > > > > >> > > > > > > > > > of > > > > > > > > >> > > > > > > > > > > data. > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > If you want to discuss most ALS in > > > APISIX, I > > > > > am > > > > > > > glad > > > > > > > > >> to > > > > > > > > >> > help > > > > > > > > >> > > > > with > > > > > > > > >> > > > > > > the > > > > > > > > >> > > > > > > > > > > design and integration solution. > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > Sheng Wu 吴晟 > > > > > > > > >> > > > > > > > > > > Twitter, wusheng1108 > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > Ming Wen <wenm...@apache.org> > > > 于2021年3月19日周五 > > > > > > > > >> 下午11:08写道: > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > yes, Prometheus is not designed for > > > > > URI-level > > > > > > > > >> metrics. > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > Does envoy ALS will send too much > data > > > and > > > > > > cause > > > > > > > > >> > performance > > > > > > > > >> > > > > > > problems? > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > And I know skywalking will supoort > > envoy > > > > > WASM, > > > > > > > is > > > > > > > > >> there > > > > > > > > >> > > > > > anything > > > > > > > > >> > > > > > > apisix > > > > > > > > >> > > > > > > > > > > can > > > > > > > > >> > > > > > > > > > > > learn from this? > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > Sheng Wu <wu.sheng.841...@gmail.com > > > > > > >于2021年3月19日 > > > > > > > > >> > 周五下午8:11写道: > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > Hi APISIX & SkyWalking community > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > I was discussing with Wei Jin > about > > > > using > > > > > > > > >> Prometheus > > > > > > > > >> > SDK > > > > > > > > >> > > > > to > > > > > > > > >> > > > > > > monitor > > > > > > > > >> > > > > > > > > > > > > URI-level metrics of > > APISIX/OpenResty. > > > > > > > > >> > > > > > > > > > > > > Wei received feedback that way of > > > using > > > > > > > > Prometheus > > > > > > > > >> > SDK > > > > > > > > >> > > > > > causing > > > > > > > > >> > > > > > > > > > > > > a significant performance impact. > > From > > > > the > > > > > > > > >> > observability > > > > > > > > >> > > > > > > perspective, > > > > > > > > >> > > > > > > > > > > > Prom > > > > > > > > >> > > > > > > > > > > > > is not good at gathering a large > > scale > > > > of > > > > > > > > >> entities' > > > > > > > > >> > > > > metrics > > > > > > > > >> > > > > > > due to > > > > > > > > >> > > > > > > > > > its > > > > > > > > >> > > > > > > > > > > > > local cache and pull mode. > > > > > > > > >> > > > > > > > > > > > > Instead, we should consider > > providing > > > a > > > > > push > > > > > > > way > > > > > > > > >> like > > > > > > > > >> > > > > Envoy > > > > > > > > >> > > > > > > ALS[1] > > > > > > > > >> > > > > > > > > > and > > > > > > > > >> > > > > > > > > > > > > Metrics Service[2] did(it does > > through > > > > > gRPC, > > > > > > > and > > > > > > > > >> > > > > SkyWalking > > > > > > > > >> > > > > > > > > > recommended > > > > > > > > >> > > > > > > > > > > > in > > > > > > > > >> > > > > > > > > > > > > the prod environment. > > > > > > > > >> > > > > > > > > > > > > If this is accepted and agreed by > > the > > > > > APISIX > > > > > > > > >> > community, we > > > > > > > > >> > > > > > > could look > > > > > > > > >> > > > > > > > > > > for > > > > > > > > >> > > > > > > > > > > > > implementation in the > > > > > > skywalking-nginx-lua[3] > > > > > > > > >> repo to > > > > > > > > >> > > > > collect > > > > > > > > >> > > > > > > > > > > > metrics(meter > > > > > > > > >> > > > > > > > > > > > > in SkyWalking) and use HTTP(in > > > > SkyWalking > > > > > > > native > > > > > > > > >> > meter > > > > > > > > >> > > > > > format) > > > > > > > > >> > > > > > > to > > > > > > > > >> > > > > > > > > > > report > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > This mail has been sent to > > > > > > > > dev@apisix.apache.org > > > > > > > > >> and > > > > > > > > >> > > > > > > > > > > > > d...@skywalking.apache.org. Please > > > reply > > > > > all > > > > > > > > when > > > > > > > > >> you > > > > > > > > >> > > > > discuss > > > > > > > > >> > > > > > > in the > > > > > > > > >> > > > > > > > > > > > > thread. > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > [1] > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/accesslog/v2/als.proto > > > > > > > > >> > > > > > > > > > > > > [2] > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/metrics/v2/metrics_service.proto > > > > > > > > >> > > > > > > > > > > > > [3] > > > > > > > > >> https://github.com/apache/skywalking-nginx-lua > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > Sheng Wu 吴晟 > > > > > > > > >> > > > > > > > > > > > > Twitter, wusheng1108 > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > -- > > > > > > > > >> > > > > > > > > > > > Thanks, > > > > > > > > >> > > > > > > > > > > > Ming Wen, Apache APISIX PMC Chair > > > > > > > > >> > > > > > > > > > > > Twitter: _WenMing > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > -- > > > > > > > > >> > > > > > > > > > Thanks, > > > > > > > > >> > > > > > > > > > Ming Wen, Apache APISIX PMC Chair > > > > > > > > >> > > > > > > > > > Twitter: _WenMing > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > -- > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > *MembPhis* > > > > > > > > >> > > > > > > > > My GitHub: https://github.com/membphis > > > > > > > > >> > > > > > > > > Apache APISIX: > > > https://github.com/apache/apisix > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > -- > > > > > > > > >> > > > > > Best regards > > > > > > > > >> > > > > > Alex Zhang > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > https://github.com/tokers > > > > > > > > >> > > > > > https://github.com/upyun/upyun-resty > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> -- > > > > > > > > >> > > > > > > > > >> *MembPhis* > > > > > > > > >> My GitHub: https://github.com/membphis > > > > > > > > >> Apache APISIX: https://github.com/apache/apisix > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Thanks, > Ming Wen, Apache APISIX PMC Chair > Twitter: _WenMing >