kotlovs commented on pull request #33154:
URL: https://github.com/apache/spark/pull/33154#issuecomment-872517103
Thanks for detailed explanation. Now I understand your problem. You create a
server inside the app and await user's requests.
The similar thing occurs when ThriftServer is started and it is reason, why
these conditions (like !isThriftServer(args.mainClass)) were added. But as I
understood, unlike ThriftServer, your case can't be checked in SparkSubmit.
I would not bind the closing of the context to k8s, even if there is not
such problem in other modes like yarn.
Instead, I would suggest to introduce another spark application arg, for
example --isServer (by default - false). Then you will pass it flag in your
spark-submit command (spark-submit --is-server ...).
And this flag will be checked when closing the spark context:
```
if (!args.isServer && !isShell(args.primaryResource) &&
!isSqlShell(args.mainClass) && !isThriftServer(args.mainClass)) {
try {
SparkContext.getActive.foreach(_.stop())
...
```
I just think it would be a more universal solution. I even already have code
adding such flag in one of my branches. I could make this PR if this solution
would be acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
