We can support both gprc and http to act as the communication protocol. But it needs to be discussed together.
On Wed, Oct 19, 2022 at 7:55 PM jianhua guo <guojhk...@gmail.com> wrote: > look good to me > > On Tue, Oct 18, 2022 at 11:27 PM Warden Wang <wangd95...@gmail.com> wrote: > >> Hi Jianhua, >> >> I have updated the sequence diagram in wiki >> https://cwiki.apache.org/confluence/display/GRIFFIN/Components+Diagram >> Could you please review it? >> >> Thanks, >> Warden >> >> William Guo <gu...@apache.org> 於 2022年10月18日 週二 晚上10:19寫道： >> >> > Hi Warden, >> > >> > I think type long is not sufficient for metrics,if it is a number, type >> > double should be more appropriate. >> > >> > Thanks, >> > William >> > >> > On Mon, Oct 17, 2022 at 10:55 PM Warden Wang <wangd95...@gmail.com> >> wrote: >> > >> > > Hi Jianhua >> > > For metrics, resp as long type is enough. >> > > >> > > Thanks >> > > Warden >> > > >> > > Eugene Law <liu...@apache.org> 於 2022年10月13日 週四 晚上9:16寫道： >> > > >> > > > +1 >> > > > >> > > > Let's clarify this. If the Griffin engine would be integrated into >> an >> > > > external system, we need to consider message protocol, such as >> protobuf >> > > or >> > > > thrift. >> > > > >> > > > On the other hand, if the Griffin engine would connect various data >> > > > engines, we only need to outline the data transfer structure. >> > > > >> > > > Thx >> > > > >> > > > On Thu, Oct 13, 2022 at 2:31 PM William Guo <gu...@apache.org> >> wrote: >> > > > >> > > > > hi, >> > > > > >> > > > > My opinion is, >> > > > > >> > > > > for external, to integrate with apache griffin, >> > > > > we will keep it open on the protocol, since we need to consider >> > > > integration >> > > > > efforts. >> > > > > >> > > > > but for internal components, >> > > > > say, to connect with different query engines(hive, spark, flink), >> we >> > > > prefer >> > > > > JDBC, which is very typical to connect to different engines as a >> hub. >> > > > > >> > > > > What do you think? >> > > > > >> > > > > Thanks, >> > > > > William >> > > > > >> > > > > On Thu, Oct 13, 2022 at 12:16 PM Edgar Joya <euj...@gmail.com> >> > wrote: >> > > > > >> > > > > > Will you be using proto files for grpc for the new architecture? >> > > > > > https://grpc.io/ >> > > > > > >> > > > > > On Wed, Oct 12, 2022 at 10:08 PM William Guo <gu...@apache.org> >> > > wrote: >> > > > > > >> > > > > > > Hi jianhua, >> > > > > > > >> > > > > > > We cannot see the architecture diagram in wiki >> > > > > > > >> > > > >> https://cwiki.apache.org/confluence/display/GRIFFIN/Components+Diagram >> > > > > > > But we can see the sequence diagram there. >> > > > > > > >> > > > > > > Could you rephrase it ? >> > > > > > > >> > > > > > > And as Eugence said, we need to make it clear which protocol >> we >> > are >> > > > > using >> > > > > > > between different components. >> > > > > > > >> > > > > > > Thanks, >> > > > > > > William >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > On Wed, Oct 12, 2022 at 9:57 PM Eugene Law <liu...@apache.org >> > >> > > > wrote: >> > > > > > > >> > > > > > > > Hi, >> > > > > > > > >> > > > > > > > Could you point out what kind of protocol definition schema >> you >> > > > used >> > > > > in >> > > > > > > > interface description? I think we should know the accurate >> > > fields' >> > > > > > range, >> > > > > > > > type in different language. >> > > > > > > > >> > > > > > > > Thx >> > > > > > > > >> > > > > > > > On Wed, Oct 12, 2022 at 7:48 PM William Guo < >> gu...@apache.org> >> > > > > wrote: >> > > > > > > > >> > > > > > > > > Thanks jianhua. >> > > > > > > > > >> > > > > > > > > Could you draw the sequence diagram in wiki >> > > > > > > > > >> > > > > > > > > >> > > > > > >> > > >> https://cwiki.apache.org/confluence/display/GRIFFIN/Components+Diagram >> > > > > > > > > >> > > > > > > > > We cannot see your attachments in email. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > William >> > > > > > > > > >> > > > > > > > > On Wed, Oct 12, 2022 at 5:58 PM jianhua guo < >> > > guojhk...@gmail.com >> > > > > >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > The core module and dispatcher's architecture is below: >> > > > > > > > > > [image: image.png] >> > > > > > > > > > 1. core module will generate the DQ sql according to the >> > > > > different >> > > > > > > > > > engine‘s syntax features，and wrapped in the submit >> request. >> > > > > > > > > > 2. dispatcher provided the restful api to accept the >> > request >> > > > from >> > > > > > the >> > > > > > > > > core >> > > > > > > > > > server. >> > > > > > > > > > 3. after received the request info, dispatcher will >> submit >> > > the >> > > > > > query >> > > > > > > to >> > > > > > > > > > the specified engine(presto, spark, hive, flink) >> > > > > > > > > > 4. dispatcher need provided a method to the core module, >> > > named >> > > > > > > > "execute" >> > > > > > > > > > is nice, and in the method, it will continue to get job >> > > status >> > > > > from >> > > > > > > > > > dispatcher,when the job is finished, then fetch the >> result >> > > from >> > > > > > > > > dispatcher, >> > > > > > > > > > then return the response to the core module. >> > > > > > > > > > The below is the sequence diagram. >> > > > > > > > > > [image: image.png] >> > > > > > > > > > [image: griffinDispatcher1.png] >> > > > > > > > > > [image: image.png] >> > > > > > > > > > Thanks for you review. >> > > > > > > > > > >> > > > > > > > > > On Wed, Oct 12, 2022 at 5:09 PM jianhua guo < >> > > > guojhk...@gmail.com >> > > > > > >> > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > >> Most looks good to me. Just one confusion, the >> > > > "RecordResponse" >> > > > > > only >> > > > > > > > > >> return long value result? >> > > > > > > > > >> >> > > > > > > > > >> On Wed, Oct 12, 2022 at 1:22 PM William Guo < >> > > gu...@apache.org >> > > > > >> > > > > > > wrote: >> > > > > > > > > >> >> > > > > > > > > >>> hi all, >> > > > > > > > > >>> >> > > > > > > > > >>> The following message from Warden, we can discuss >> > > interaction >> > > > > > > between >> > > > > > > > > >>> core >> > > > > > > > > >>> and dispatcher module here. >> > > > > > > > > >>> ==================== >> > > > > > > > > >>> I have drafted the communication messages between core >> > and >> > > > > > > dispatcher >> > > > > > > > > as >> > > > > > > > > >>> below, >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > > >>> // submit sql >> > > > > > > > > >>> execute(RecordRequest) return RecordResponse; >> > > > > > > > > >>> >> > > > > > > > > >>> RecordRequest { >> > > > > > > > > >>> String recordSql; >> > > > > > > > > >>> Enum engine; // Spark,Hive,Presto,etc. >> > > > > > > > > >>> String owner; >> > > > > > > > > >>> Integer maxRetryCount; >> > > > > > > > > >>> } >> > > > > > > > > >>> >> > > > > > > > > >>> RecordResponse { >> > > > > > > > > >>> Integer code; >> > > > > > > > > >>> Long value; >> > > > > > > > > >>> Enum errorCode; // if code != 200, please tell us >> what >> > > > > happened: >> > > > > > > > > >>> // 1、 recordSql syntax error >> > > > > > > > > >>> // 2、 internal error, dispatcher self is >> crashed >> > > > > > > > > >>> // 3、 external error, target engine is crashed >> > when >> > > > > > > > dispatcher >> > > > > > > > > >>> call,etc >> > > > > > > > > >>> Exception ex; // error detail info >> > > > > > > > > >>> } >> > > > > > > > > >>> >> > > > > > > > > >>> // validate sql syntax >> > > > > > > > > >>> validateSQL(CheckRequest) return CheckResponse; >> > > > > > > > > >>> >> > > > > > > > > >>> ValidateSQLRequest { >> > > > > > > > > >>> String recordSql; >> > > > > > > > > >>> Enum engine; // Spark,Hive,Presto,etc. >> > > > > > > > > >>> } >> > > > > > > > > >>> >> > > > > > > > > >>> ValidateSQLResponse { >> > > > > > > > > >>> Integer code; >> > > > > > > > > >>> Enum errorCode; // if code != 200, please tell us >> what >> > > > > happened: >> > > > > > > > > >>> // 1、 recordSql syntax error >> > > > > > > > > >>> // 2、 internal error, dispatcher self is >> crashed >> > > > > > > > > >>> // 3、 external error, target engine is crashed >> > when >> > > > > > > > dispatcher >> > > > > > > > > >>> call,etc >> > > > > > > > > >>> Exception ex; // error detail info >> > > > > > > > > >>> } >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > > >>> Could you please review it? Give us your feedback. >> > > > > > > > > >>> >> > > > > > > > > >>> ================== >> > > > > > > > > >>> >> > > > > > > > > >> >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Edgar Joya >> > > > > > 980.259.0683 >> > > > > > euj...@gmail.com >> > > > > > @eujc21 >> > > > > > >> > > > > >> > > > >> > > >> > >> >
