Hi wanxun, I'll write document for `external.config.location`.
Still not clear why Livy server rejects the other jobs, except the first job. Could you explain in detail? Thanks, Kevin On Mon, Sep 23, 2019 at 3:50 PM 万昆 <[email protected]> wrote: > > > > Hi, Kevin > > > Thanks for your reply. > Could you help with the documentation about parameter > external.config.location ？ > > > When the Griffin submits some jobs with the same session name at the > same time, Livy server will reject the other jobs, except the first job, > because the session is already started. So I think this problem is not > related to where the configuration files are located at. > > > At 2019-09-18 14:17:00, "Kevin Yao" <[email protected]> wrote: > >Seems pr closed. I cannot comment it. > >I agree with you. we should have document to introduce such function for > >users. > > > >Thanks, > >Kevin > > > >On Wed, Sep 18, 2019 at 12:38 PM William Guo <[email protected]> wrote: > > > >> hi Kevin, > >> > >> the PR is https://github.com/apache/griffin/pull/534 , Could you review > >> the > >> PR again? > >> > >> external.config.location need more documentation for users. > >> > >> > >> Thanks, > >> William > >> > >> On Wed, Sep 18, 2019 at 10:27 AM Kevin Yao <[email protected]> > wrote: > >> > >> > Hi William, wankun, > >> > > >> > Actually, I don't think it's a bug for Griffin. `sparkProperties.json` > >> is a > >> > config file which can be replaced by user in external location. We > have > >> > already supported such function with `external.config.location` in > >> > `application.properties` file. > >> > > >> > Thanks, > >> > Kevin > >> > > >> > On Wed, Sep 18, 2019 at 7:30 AM ASF GitHub Bot (Jira) < > [email protected]> > >> > wrote: > >> > > >> > > > >> > > [ > >> > > > >> > > >> > https://issues.apache.org/jira/browse/GRIFFIN-290?focusedWorklogId=314053&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-314053 > >> > > ] > >> > > > >> > > ASF GitHub Bot logged work on GRIFFIN-290: > >> > > ------------------------------------------ > >> > > > >> > > Author: ASF GitHub Bot > >> > > Created on: 17/Sep/19 23:29 > >> > > Start Date: 17/Sep/19 23:29 > >> > > Worklog Time Spent: 10m > >> > > Work Description: asfgit commented on pull request #534: > >> > > [GRIFFIN-290]Fix bug for submitting job to livy > >> > > URL: https://github.com/apache/griffin/pull/534 > >> > > > >> > > > >> > > > >> > > > >> > > ---------------------------------------------------------------- > >> > > This is an automated message from the Apache Git Service. > >> > > To respond to the message, please log on to GitHub and use the > >> > > URL above to go to the specific comment. > >> > > > >> > > For queries about this service, please contact Infrastructure at: > >> > > [email protected] > >> > > > >> > > > >> > > Issue Time Tracking > >> > > ------------------- > >> > > > >> > > Worklog Id: (was: 314053) > >> > > Time Spent: 50m (was: 40m) > >> > > > >> > > > Fix bug for submitting job to livy > >> > > > ---------------------------------- > >> > > > > >> > > > Key: GRIFFIN-290 > >> > > > URL: > >> https://issues.apache.org/jira/browse/GRIFFIN-290 > >> > > > Project: Griffin > >> > > > Issue Type: Task > >> > > > Reporter: wan kun > >> > > > Priority: Major > >> > > > Fix For: 0.6.0 > >> > > > > >> > > > Time Spent: 50m > >> > > > Remaining Estimate: 0h > >> > > > > >> > > > When griffin submit multiple DQ jobs to livy, the http parameter > >> `name` > >> > > is always griffin. > >> > > > So livy will reject them. > >> > > > job request : > >> > > > > >> > > > {quote}[owner: null, request: [proxyUser: None, file: > >> > > hdfs://nameservice-standby/user/kun.wan/measure-0.6.0-SNAPSHOT.jar, > >> > > > args: { > >> > > > "spark" : > >> > > > Unknown macro: \{ "log.level" } > >> > > > , > >> > > > "sinks" : [ > >> > > > Unknown macro: \{ .... } > >> > > > ], > >> > > > "griffin.checkpoint" : [ ] > >> > > > },{ > >> > > > "measure.type" : "griffin", > >> > > > "id" : 5202, > >> > > > "name" : "spu_null_check", > >> > > > "owner" : "test", > >> > > > "description" : "check null value for store and category", > >> > > > "deleted" : false, > >> > > > "timestamp" : 1568195100000, > >> > > > "dq.type" : "PROFILING", > >> > > > "sinks" : [ "ELASTICSEARCH", "HDFS" ], > >> > > > "process.type" : "BATCH", > >> > > > "rule.description" : > >> > > > , > >> > > > "data.sources" : [ > >> > > > Unknown macro: \{ .... } > >> > > > ], > >> > > > "evaluate.rule" : > >> > > > , > >> > > > "measure.type" : "griffin" > >> > > > },raw,raw, driverMemory: 1g, executorMemory: 6g, executorCores: > 2, > >> > > numExecutors: 6, queue: root.users.kun_dot_wan, > {color:#ff0000}*name: > >> > > griffin*{color}]] > >> > > > {quote} > >> > > > > >> > > > livy Response : > >> > > > {quote}400 Bad Request > >> > > > [Date:"Thu, 12 Sep 2019 10:00:00 GMT", > >> > > Content-Type:"application/json;charset=utf-8", Content-Length:"47", > >> > > Server:"Jetty(9.3.24.v20180605)"] > >> > > > {"msg":"Duplicate session name: Some(griffin)"} > >> > > > {quote} > >> > > > >> > > > >> > > > >> > > -- > >> > > This message was sent by Atlassian Jira > >> > > (v8.3.4#803005) > >> > > > >> > > >> >
