suheng.cloud created LIVY-428: --------------------------------- Summary: Does livy support concurrency for centain spark session? Key: LIVY-428 URL: https://issues.apache.org/jira/browse/LIVY-428 Project: Livy Issue Type: Improvement Components: API Affects Versions: 0.4 Environment: livy 0.4 spark2.2.0-cloudera1 Reporter: suheng.cloud Attachments: spark-fair.png
To use spark sql concurrency for internal fair schedule in [post] /sessions/{id}/statements use param "conf": {"spark.scheduler.mode":"FAIR","spark.scheduler.allocation.file":"/home/.../spark-schedule.xml"} spark-schedule.xml is as follows: <allocations> <pool name="FAIR"> <schedulingMode>FAIR</schedulingMode> </pool> </allocations> Log shows tow jobs surely be added into fair pool: INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_10.0 tasks to pool FAIR ... INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_11.0 tasks to pool FAIR But from spark web console the second job starts only when the first finished. I also tried with zeppelin, which concurrency works well with spark but not work in livy. Did I miss some configuration? Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)