Re: Ctrl - left and right now working in Spark Shell in Windows 10
This won't be related to Spark, but rather your shell or terminal program. On Tue, Nov 1, 2022 at 1:57 PM Salil Surendran wrote: > I installed Spark on Windows 10. Everything works fine except for the Ctrl > - left and Ctrl - right keys which doesn't move a word but just a > character. How do I fix this or find out what are the correct bindings to > move a word in Spark Shell > > -- > Thanks, > Salil > "The surest sign that intelligent life exists elsewhere in the universe is > that none of it has tried to contact us." >
Ctrl - left and right now working in Spark Shell in Windows 10
I installed Spark on Windows 10. Everything works fine except for the Ctrl - left and Ctrl - right keys which doesn't move a word but just a character. How do I fix this or find out what are the correct bindings to move a word in Spark Shell -- Thanks, Salil "The surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us."
Re: The Dataset unit test is much slower than the RDD unit test (in Scala)
Which Spark version are you using? SPARK-36444[1] and SPARK-38138[2] may be related, please test w/ the patched version or disable DPP by setting spark.sql.optimizer.dynamicPartitionPruning.enabled=false to see if it helps. [1] https://issues.apache.org/jira/browse/SPARK-36444 [2] https://issues.apache.org/jira/browse/SPARK-38138 Thanks, Cheng Pan On Nov 2, 2022 at 00:14:34, Enrico Minack wrote: > Hi Tanin, > > running your test with option "spark.sql.planChangeLog.level" set to > "info" or "warn" (depending on your Spark log level) will show you > insights into the planning (which rules are applied, how long rules > take, how many iterations are done). > > Hoping this helps, > Enrico > > > Am 25.10.22 um 21:54 schrieb Tanin Na Nakorn: > > Hi All, > > > Our data job is very complex (e.g. 100+ joins), and we have switched > > from RDD to Dataset recently. > > > We've found that the unit test takes much longer. We profiled it and > > have found that it's the planning phase that is slow, not execution. > > > I wonder if anyone has encountered this issue before and if there's a > > way to make the planning phase faster (e.g. maybe disabling certain > > optimizers). > > > Any thoughts or input would be appreciated. > > > Thank you, > > Tanin > > > > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: The Dataset unit test is much slower than the RDD unit test (in Scala)
Hi Tanin, running your test with option "spark.sql.planChangeLog.level" set to "info" or "warn" (depending on your Spark log level) will show you insights into the planning (which rules are applied, how long rules take, how many iterations are done). Hoping this helps, Enrico Am 25.10.22 um 21:54 schrieb Tanin Na Nakorn: Hi All, Our data job is very complex (e.g. 100+ joins), and we have switched from RDD to Dataset recently. We've found that the unit test takes much longer. We profiled it and have found that it's the planning phase that is slow, not execution. I wonder if anyone has encountered this issue before and if there's a way to make the planning phase faster (e.g. maybe disabling certain optimizers). Any thoughts or input would be appreciated. Thank you, Tanin - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
