+1 Let's clarify this. If the Griffin engine would be integrated into an external system, we need to consider message protocol, such as protobuf or thrift.
On the other hand, if the Griffin engine would connect various data engines, we only need to outline the data transfer structure. Thx On Thu, Oct 13, 2022 at 2:31 PM William Guo <[email protected]> wrote: > hi, > > My opinion is, > > for external, to integrate with apache griffin, > we will keep it open on the protocol, since we need to consider integration > efforts. > > but for internal components, > say, to connect with different query engines(hive, spark, flink), we prefer > JDBC, which is very typical to connect to different engines as a hub. > > What do you think? > > Thanks, > William > > On Thu, Oct 13, 2022 at 12:16 PM Edgar Joya <[email protected]> wrote: > > > Will you be using proto files for grpc for the new architecture? > > https://grpc.io/ > > > > On Wed, Oct 12, 2022 at 10:08 PM William Guo <[email protected]> wrote: > > > > > Hi jianhua, > > > > > > We cannot see the architecture diagram in wiki > > > https://cwiki.apache.org/confluence/display/GRIFFIN/Components+Diagram > > > But we can see the sequence diagram there. > > > > > > Could you rephrase it ? > > > > > > And as Eugence said, we need to make it clear which protocol we are > using > > > between different components. > > > > > > Thanks, > > > William > > > > > > > > > > > > On Wed, Oct 12, 2022 at 9:57 PM Eugene Law <[email protected]> wrote: > > > > > > > Hi, > > > > > > > > Could you point out what kind of protocol definition schema you used > in > > > > interface description? I think we should know the accurate fields' > > range, > > > > type in different language. > > > > > > > > Thx > > > > > > > > On Wed, Oct 12, 2022 at 7:48 PM William Guo <[email protected]> > wrote: > > > > > > > > > Thanks jianhua. > > > > > > > > > > Could you draw the sequence diagram in wiki > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/GRIFFIN/Components+Diagram > > > > > > > > > > We cannot see your attachments in email. > > > > > > > > > > Thanks, > > > > > William > > > > > > > > > > On Wed, Oct 12, 2022 at 5:58 PM jianhua guo <[email protected]> > > > wrote: > > > > > > > > > > > The core module and dispatcher's architecture is below: > > > > > > [image: image.png] > > > > > > 1. core module will generate the DQ sql according to the > different > > > > > > engine‘s syntax features，and wrapped in the submit request. > > > > > > 2. dispatcher provided the restful api to accept the request from > > the > > > > > core > > > > > > server. > > > > > > 3. after received the request info, dispatcher will submit the > > query > > > to > > > > > > the specified engine(presto, spark, hive, flink) > > > > > > 4. dispatcher need provided a method to the core module, named > > > > "execute" > > > > > > is nice, and in the method, it will continue to get job status > from > > > > > > dispatcher,when the job is finished, then fetch the result from > > > > > dispatcher, > > > > > > then return the response to the core module. > > > > > > The below is the sequence diagram. > > > > > > [image: image.png] > > > > > > [image: griffinDispatcher1.png] > > > > > > [image: image.png] > > > > > > Thanks for you review. > > > > > > > > > > > > On Wed, Oct 12, 2022 at 5:09 PM jianhua guo <[email protected] > > > > > > wrote: > > > > > > > > > > > >> Most looks good to me. Just one confusion, the "RecordResponse" > > only > > > > > >> return long value result? > > > > > >> > > > > > >> On Wed, Oct 12, 2022 at 1:22 PM William Guo <[email protected]> > > > wrote: > > > > > >> > > > > > >>> hi all, > > > > > >>> > > > > > >>> The following message from Warden, we can discuss interaction > > > between > > > > > >>> core > > > > > >>> and dispatcher module here. > > > > > >>> ==================== > > > > > >>> I have drafted the communication messages between core and > > > dispatcher > > > > > as > > > > > >>> below, > > > > > >>> > > > > > >>> > > > > > >>> // submit sql > > > > > >>> execute(RecordRequest) return RecordResponse; > > > > > >>> > > > > > >>> RecordRequest { > > > > > >>> String recordSql; > > > > > >>> Enum engine; // Spark,Hive,Presto,etc. > > > > > >>> String owner; > > > > > >>> Integer maxRetryCount; > > > > > >>> } > > > > > >>> > > > > > >>> RecordResponse { > > > > > >>> Integer code; > > > > > >>> Long value; > > > > > >>> Enum errorCode; // if code != 200, please tell us what > happened: > > > > > >>> // 1、 recordSql syntax error > > > > > >>> // 2、 internal error, dispatcher self is crashed > > > > > >>> // 3、 external error, target engine is crashed when > > > > dispatcher > > > > > >>> call,etc > > > > > >>> Exception ex; // error detail info > > > > > >>> } > > > > > >>> > > > > > >>> // validate sql syntax > > > > > >>> validateSQL(CheckRequest) return CheckResponse; > > > > > >>> > > > > > >>> ValidateSQLRequest { > > > > > >>> String recordSql; > > > > > >>> Enum engine; // Spark,Hive,Presto,etc. > > > > > >>> } > > > > > >>> > > > > > >>> ValidateSQLResponse { > > > > > >>> Integer code; > > > > > >>> Enum errorCode; // if code != 200, please tell us what > happened: > > > > > >>> // 1、 recordSql syntax error > > > > > >>> // 2、 internal error, dispatcher self is crashed > > > > > >>> // 3、 external error, target engine is crashed when > > > > dispatcher > > > > > >>> call,etc > > > > > >>> Exception ex; // error detail info > > > > > >>> } > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> Could you please review it? Give us your feedback. > > > > > >>> > > > > > >>> ================== > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > -- > > Edgar Joya > > 980.259.0683 > > [email protected] > > @eujc21 > > >
