[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22919 Let me go ahead with documenting one then tomorrow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22919 True. but it's a bit difficult to say it's a downside because `-i` has a quite major issue as described above - user's code suddenly does not work with implicit method (like symbol `'id` or `.toDF` which are pretty common) in minor release bumpup. Also, looks the only reason why changed `:load` to `:paste` in `-i` option is a bug (SI-7898). For documentation, I can mention we can use Scala shell flags. Looks @cloud-fan is going to leave this JIRA as a known issue so I guess we are good. Maybe I can leave an additional note in migration guide as well if we're not going ahead to fix it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22919 This is weird. The most obvious downside is that there's no way to get the new `-i` functionality in case anyone needs it. A blurb in `--help` or even a release note explaining the changed behavior would be good enough, I think. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22919 Tough call. At least it's worth documenting that this behavior changed because so did the Scala shell's behavior, and I'd support that. I'd also support supporting both, if there's no real downside to that option? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22919 Thabks, @dbtsai. Also adding @vanzin and @srowen wdyt about -i options at 2.4.x and 3.0.0? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22919 Thanks, @dbtsai . That sounds reasonable and helpful, too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/22919 I'm also on @cloud-fan's side---we should keep it consistent with the upstream Scala Shell. However, we should document it on `./bin/spark-shell --help`, so when a user complains or files a ticket, we can refer them to the doc. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22919 +1 for @HyukjinKwon 's opinion . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22919 Also, the difference introduced between `-i` and `-I` sounds rather a bug fix ([SI-7898](https://issues.scala-lang.org/browse/SI-7898)), which already exists in previous shell. I ended up with thinking that it might be more important that end users upgrade Spark and face such issues than syncing it to Scala 2.11.12 shell at Spark 2.4.x. I don't superduper strongly feel about it but it was my conclusion. I will defer to guys here about going ahead in this way or not. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22919 Note that `python` shell also takes file but `pyspark` shell doesn't: ``` $ cat tmp.py print "a" ``` ``` $ python2 tmp.py a ``` ``` $ ./bin/pyspark tmp.py Running python applications through 'pyspark' is not supported as of Spark 2.0. Use ./bin/spark-submit ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22919 I was actually thinking that hard about that. My conclusion was that basically we should but `spark-shell` is a Spark entry point where we make a release - so I thought it's better to keep it in 2.4.0. In 3.0.0, actually I think we should disallow both `-i` and `-I` like `pyspark` or `sparkr` shell (we allowed file before but not anymore). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22919 personally I think Spark Shell should be consistent with the upstream Scala Shell, otherwise we may get another ticket complaining why we didn't follow... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22919 Yup. That's what I thought. and treat them as same --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22919: [SPARK-25906][SHELL] Restores '-i' option's behaviour in...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22919 so we would support both `-i` and `-I` in 2.4? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
