GitHub user lvdongr opened a pull request: https://github.com/apache/spark/pull/20356
[SPARK-23185][SQL] Make the configuration "spark.default.parallelism" can be changed on each SQL session to decrease empty files ## What changes were proposed in this pull request? Make the configuration "spark.default.parallelism" can be changed on each SQL session to decrease empty files. When execute "insert into ... values ...", many empty files will be generated.We can change the configuration "spark.default.parallelism" to decrease the number of empty files.But there are many occasions that we want to chang the configuration during each session so as not to influence other sql sentences, like we may use thrift server to excute many sql sentences on a SQL session. ## How was this patch tested? unit tests, manual tests Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lvdongr/spark SPARK-23185 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20356.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20356 ---- commit 01af8ce69afeade8bb034c6965de0f3738f12fd5 Author: lvdongr <lv.dongdong@...> Date: 2017-03-08T04:09:40Z [SPARK-19863][DStream] Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application has been successfully created. commit b6daeec664d757999e257e56fed3844db51515e2 Author: lvdongr <lv.dongdong@...> Date: 2017-03-11T06:35:57Z Merge remote-tracking branch 'apache/master' commit e0e47b1da93b90210e44abc6e90655d3028555ec Author: lvdongr <lv.dongdong@...> Date: 2017-04-12T07:20:01Z Merge remote-tracking branch 'apache/master' commit f4ab88111c5b8e9700eacc1acfa3858aed45124e Author: lvdongr <lv.dongdong@...> Date: 2017-07-27T01:54:56Z isklakldsng branch 'apache/master' commit 463e570f9e05f785834e27bd535cfbb3b7cb7dfb Author: lvdongr <lv.dongdong@...> Date: 2017-07-27T12:09:47Z Merge remote-tracking branch 'apache/master' commit 0e1b7f6d8e436ca243f78e3cbf064f591557b6c0 Author: lvdongr <lv.dongdong@...> Date: 2017-07-28T01:34:48Z Merge remote-tracking branch 'apache/master' commit 9a9972125ae8f7d90f5567f5b561f2c0ca16cfe7 Author: lvdongr <lv.dongdong@...> Date: 2017-07-28T02:50:23Z refresh the master branch for kafkaconsumer commit 637900b576b8c4d9e04a808a078e481a99751d03 Author: lvdongr <lv.dongdong@...> Date: 2017-07-31T03:08:29Z Merge remote-tracking branch 'apache/master' commit 04aafed076cb704a100eb7dc45b5cfda6438193b Author: lvdongr <lv.dongdong@...> Date: 2017-08-17T11:41:58Z Merge remote-tracking branch 'apache/master' commit 9f90ab5356b74dfc63dc9c80ff336ef2c2847e72 Author: root <root@...> Date: 2017-11-10T03:32:54Z Merge branch 'master' of https://github.com/apache/spark commit 8b94711b7fb6cfa72aa06d9e009b73b73ccda36f Author: root <root@...> Date: 2017-11-13T00:56:22Z Merge branch 'master' of https://github.com/apache/spark commit 70699e3d80d853f7105d967544378c5c342d2ce6 Author: 10171592 <lv.dongdong@...> Date: 2017-12-07T03:13:24Z Merge remote-tracking branch 'apache/master' commit 9e7c0c7d0f8bae30bc07abbedf4c110ec82f1cf3 Author: root <root@...> Date: 2017-12-07T05:48:04Z Merge remote-tracking branch 'apache/master' commit 393730415bcebdef125364be3eb3a64320cac3c9 Author: root <root@...> Date: 2018-01-09T03:16:36Z Merge branch 'master' of https://github.com/lvdongr/spark commit 5db407930d4802b6075036961688192a3039d95a Author: root <root@...> Date: 2018-01-09T03:30:43Z Merge branch 'master' of https://github.com/apache/spark commit 46672ddaf53b9ed1e97e404753fa14bd3406821a Author: 10171592 <lv.dongdong@...> Date: 2018-01-22T08:20:47Z Merge remote-tracking branch 'apache/master' commit 884eaee9f2d7782bceae73806da9b65f1119977e Author: 10171592 <lv.dongdong@...> Date: 2018-01-22T09:30:14Z Merge branch 'master' of https://github.com/lvdongr/spark commit 49641920727f426e88ac32a9c1381f7876eaf7c9 Author: 10171592 <lv.dongdong@...> Date: 2018-01-23T02:57:54Z Merge remote-tracking branch 'apache/master' commit e1aeff8c0cb1358d0c77b0e729ecdfd1a07313dc Author: lvdongr <lv.dongdong@...> Date: 2018-01-23T03:18:28Z [SPARK-23185][SQL] Make the configuration "spark.default.parallelism" can be changed on each SQL session to decrease empty files ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org