[GitHub] [spark] ajithme commented on issue #27267: [WIP][SPARK-30556][SQL] Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext
ajithme commented on issue #27267: [WIP][SPARK-30556][SQL] Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext URL: https://github.com/apache/spark/pull/27267#issuecomment-575728914 > Closely related to #27266? Currently, `org.apache.spark.sql.execution.SubqueryExec#executionContext` has a hardcoded size of 16 threads which makes writing UT difficult so i plan to make it configurable + fix subquery thread bug hence created a new pull request compared to broadcast bug (which is independent of this fix) in https://github.com/apache/spark/pull/27266 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27267: [WIP][SPARK-30556][SQL] Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext
ajithme commented on issue #27267: [WIP][SPARK-30556][SQL] Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext URL: https://github.com/apache/spark/pull/27267#issuecomment-576351616 > IMO the thread pools should not be a big issue. The subquery is guaranteed to be executed on a different thread (you can even add an assert for this). You just set some unique property on the local properties (value should also be unique), construct the something that contains a broadcast join, use an accumulator that you modify using either a UDF (easy) or a dataset operation. Agree. But with a pool size of 16, i will have to ensure all 16 threads are used at-least once and are alive to reproduce this issue and not to make that test flaky. Its lot easier if i can set the pool size to 1 in test to reproduce. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org