Hi Jianhua For metrics, resp as long type is enough. Thanks Warden
Eugene Law <[email protected]> 於 2022年10月13日 週四 晚上9:16寫道： > +1 > > Let's clarify this. If the Griffin engine would be integrated into an > external system, we need to consider message protocol, such as protobuf or > thrift. > > On the other hand, if the Griffin engine would connect various data > engines, we only need to outline the data transfer structure. > > Thx > > On Thu, Oct 13, 2022 at 2:31 PM William Guo <[email protected]> wrote: > > > hi, > > > > My opinion is, > > > > for external, to integrate with apache griffin, > > we will keep it open on the protocol, since we need to consider > integration > > efforts. > > > > but for internal components, > > say, to connect with different query engines(hive, spark, flink), we > prefer > > JDBC, which is very typical to connect to different engines as a hub. > > > > What do you think? > > > > Thanks, > > William > > > > On Thu, Oct 13, 2022 at 12:16 PM Edgar Joya <[email protected]> wrote: > > > > > Will you be using proto files for grpc for the new architecture? > > > https://grpc.io/ > > > > > > On Wed, Oct 12, 2022 at 10:08 PM William Guo <[email protected]> wrote: > > > > > > > Hi jianhua, > > > > > > > > We cannot see the architecture diagram in wiki > > > > > https://cwiki.apache.org/confluence/display/GRIFFIN/Components+Diagram > > > > But we can see the sequence diagram there. > > > > > > > > Could you rephrase it ? > > > > > > > > And as Eugence said, we need to make it clear which protocol we are > > using > > > > between different components. > > > > > > > > Thanks, > > > > William > > > > > > > > > > > > > > > > On Wed, Oct 12, 2022 at 9:57 PM Eugene Law <[email protected]> > wrote: > > > > > > > > > Hi, > > > > > > > > > > Could you point out what kind of protocol definition schema you > used > > in > > > > > interface description? I think we should know the accurate fields' > > > range, > > > > > type in different language. > > > > > > > > > > Thx > > > > > > > > > > On Wed, Oct 12, 2022 at 7:48 PM William Guo <[email protected]> > > wrote: > > > > > > > > > > > Thanks jianhua. > > > > > > > > > > > > Could you draw the sequence diagram in wiki > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/GRIFFIN/Components+Diagram > > > > > > > > > > > > We cannot see your attachments in email. > > > > > > > > > > > > Thanks, > > > > > > William > > > > > > > > > > > > On Wed, Oct 12, 2022 at 5:58 PM jianhua guo <[email protected] > > > > > > wrote: > > > > > > > > > > > > > The core module and dispatcher's architecture is below: > > > > > > > [image: image.png] > > > > > > > 1. core module will generate the DQ sql according to the > > different > > > > > > > engine‘s syntax features，and wrapped in the submit request. > > > > > > > 2. dispatcher provided the restful api to accept the request > from > > > the > > > > > > core > > > > > > > server. > > > > > > > 3. after received the request info, dispatcher will submit the > > > query > > > > to > > > > > > > the specified engine(presto, spark, hive, flink) > > > > > > > 4. dispatcher need provided a method to the core module, named > > > > > "execute" > > > > > > > is nice, and in the method, it will continue to get job status > > from > > > > > > > dispatcher,when the job is finished, then fetch the result from > > > > > > dispatcher, > > > > > > > then return the response to the core module. > > > > > > > The below is the sequence diagram. > > > > > > > [image: image.png] > > > > > > > [image: griffinDispatcher1.png] > > > > > > > [image: image.png] > > > > > > > Thanks for you review. > > > > > > > > > > > > > > On Wed, Oct 12, 2022 at 5:09 PM jianhua guo < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > >> Most looks good to me. Just one confusion, the > "RecordResponse" > > > only > > > > > > >> return long value result? > > > > > > >> > > > > > > >> On Wed, Oct 12, 2022 at 1:22 PM William Guo <[email protected] > > > > > > wrote: > > > > > > >> > > > > > > >>> hi all, > > > > > > >>> > > > > > > >>> The following message from Warden, we can discuss interaction > > > > between > > > > > > >>> core > > > > > > >>> and dispatcher module here. > > > > > > >>> ==================== > > > > > > >>> I have drafted the communication messages between core and > > > > dispatcher > > > > > > as > > > > > > >>> below, > > > > > > >>> > > > > > > >>> > > > > > > >>> // submit sql > > > > > > >>> execute(RecordRequest) return RecordResponse; > > > > > > >>> > > > > > > >>> RecordRequest { > > > > > > >>> String recordSql; > > > > > > >>> Enum engine; // Spark,Hive,Presto,etc. > > > > > > >>> String owner; > > > > > > >>> Integer maxRetryCount; > > > > > > >>> } > > > > > > >>> > > > > > > >>> RecordResponse { > > > > > > >>> Integer code; > > > > > > >>> Long value; > > > > > > >>> Enum errorCode; // if code != 200, please tell us what > > happened: > > > > > > >>> // 1、 recordSql syntax error > > > > > > >>> // 2、 internal error, dispatcher self is crashed > > > > > > >>> // 3、 external error, target engine is crashed when > > > > > dispatcher > > > > > > >>> call,etc > > > > > > >>> Exception ex; // error detail info > > > > > > >>> } > > > > > > >>> > > > > > > >>> // validate sql syntax > > > > > > >>> validateSQL(CheckRequest) return CheckResponse; > > > > > > >>> > > > > > > >>> ValidateSQLRequest { > > > > > > >>> String recordSql; > > > > > > >>> Enum engine; // Spark,Hive,Presto,etc. > > > > > > >>> } > > > > > > >>> > > > > > > >>> ValidateSQLResponse { > > > > > > >>> Integer code; > > > > > > >>> Enum errorCode; // if code != 200, please tell us what > > happened: > > > > > > >>> // 1、 recordSql syntax error > > > > > > >>> // 2、 internal error, dispatcher self is crashed > > > > > > >>> // 3、 external error, target engine is crashed when > > > > > dispatcher > > > > > > >>> call,etc > > > > > > >>> Exception ex; // error detail info > > > > > > >>> } > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> Could you please review it? Give us your feedback. > > > > > > >>> > > > > > > >>> ================== > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Edgar Joya > > > 980.259.0683 > > > [email protected] > > > @eujc21 > > > > > >
