[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user rxin commented on the issue: https://github.com/apache/spark/pull/22332 Thanks guys. On Thu, Sep 6, 2018 at 2:12 AM Hyukjin Kwon wrote: > Thanks, @wmellouli . > > — > You are receiving this because you were

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22332 Thanks, @wmellouli. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user wmellouli commented on the issue: https://github.com/apache/spark/pull/22332 PR closed: we can use select to add new columns in a user-defined position. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22332 If that's easily worked around, let's not add this one. There are too many APIs open now and we should rather try to reduce them. ---

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user wmellouli commented on the issue: https://github.com/apache/spark/pull/22332 @HyukjinKwon even instead of using the actual method `withColumn(colName: String, col: Column)` we can just add a column and select. The idea from this PR is to add more power/flexibility to

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22332 > What you suggested does not manage replacing existing column content. I think we can still just add a column and select ... no? ---

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user wmellouli commented on the issue: https://github.com/apache/spark/pull/22332 @HyukjinKwon Thank you for your review. To answer to your question about using `select`, take a look at my explaination [here](https://github.com/apache/spark/pull/22332#issuecomment-418526562)

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user wmellouli commented on the issue: https://github.com/apache/spark/pull/22332 @jaceklaskowski I refactored with what you suggested in your review. Let me know what you think. --- - To unsubscribe,

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22332 Can't we simply `select` after the the column is added? I wouldn't add this as well - it can look confusing to be honest IMO. ---

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user maropu commented on the issue: https://github.com/apache/spark/pull/22332 I also can't find a strong reason to append a new API in `Dataset`... btw, to add a new API there, you'd be better to discuss in jira before making a pr, I think. cc: @rxin @cloud-fan @HyukjinKwon

[GitHub] spark issue #22332: [SPARK-25333][SQL] Ability add new columns in Dataset in...

Github user wmellouli commented on the issue: https://github.com/apache/spark/pull/22332 @mgaido91 Thank you for your suggestion, I updated the PR name, description and sources with a new version using a parameter `atPosition` instead of a flag `atTheEnd`. Let me know what you think