[jira] [Created] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-15 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-25732: --- Summary: Allow specifying a keytab/principal for proxy user for token renewal Key: SPARK-25732 URL: https://issues.apache.org/jira/browse/SPARK-25732 Project: Spark

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-15 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649917#comment-16649917 ] Marco Gaido commented on SPARK-25732: - cc [~vanzin] [~tgraves] [~jerryshao] [~mridul

[jira] [Commented] (SPARK-25728) SPIP: Structured Intermediate Representation (Tungsten IR) for generating Java code

2018-10-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651480#comment-16651480 ] Marco Gaido commented on SPARK-25728: - Thanks [~kiszk]. I will check it ASAP, thanks

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651800#comment-16651800 ] Marco Gaido commented on SPARK-25732: - [~tgraves] I think they can be reused, the po

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651860#comment-16651860 ] Marco Gaido commented on SPARK-25732: - [~tgraves] yes, exactly it is what I am refer

[jira] [Created] (SPARK-25758) Deprecate BisectingKMeans compute cost

2018-10-17 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-25758: --- Summary: Deprecate BisectingKMeans compute cost Key: SPARK-25758 URL: https://issues.apache.org/jira/browse/SPARK-25758 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25758) Deprecate BisectingKMeans compute cost

2018-10-17 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653641#comment-16653641 ] Marco Gaido commented on SPARK-25758: - cc [~cloud_fan] [~srowen] [~holdenkarau]. Thi

[jira] [Created] (SPARK-25764) Avoid usage of deprecated methods in examples for BisectingKMeans

2018-10-18 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-25764: --- Summary: Avoid usage of deprecated methods in examples for BisectingKMeans Key: SPARK-25764 URL: https://issues.apache.org/jira/browse/SPARK-25764 Project: Spark

[jira] [Created] (SPARK-25765) Add trainingCost to BisectingKMeans summary

2018-10-18 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-25765: --- Summary: Add trainingCost to BisectingKMeans summary Key: SPARK-25765 URL: https://issues.apache.org/jira/browse/SPARK-25765 Project: Spark Issue Type: Improve

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655366#comment-16655366 ] Marco Gaido commented on SPARK-25767: - I tried on current master branch but I wasn't

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655418#comment-16655418 ] Marco Gaido commented on SPARK-25767: - It is interesting, I can reproduce with the J

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655440#comment-16655440 ] Marco Gaido commented on SPARK-25767: - So I tracked down the issue. The problem is t

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655530#comment-16655530 ] Marco Gaido commented on SPARK-25767: - Your conversion of a Java array in a Scala Se

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-19 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656861#comment-16656861 ] Marco Gaido commented on SPARK-25767: - I think it is a bug (thanks for reporting thi

[jira] [Commented] (SPARK-25829) Duplicated map keys are not handled consistently

2018-10-25 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663553#comment-16663553 ] Marco Gaido commented on SPARK-25829: - I think the main issue is that since this is

[jira] [Created] (SPARK-25838) Remove formatVersion from Saveable

2018-10-25 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-25838: --- Summary: Remove formatVersion from Saveable Key: SPARK-25838 URL: https://issues.apache.org/jira/browse/SPARK-25838 Project: Spark Issue Type: Task C

[jira] [Commented] (SPARK-25863) java.lang.UnsupportedOperationException: empty.max at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.updateAndGetCompilationStats(CodeGenerator.scala

2018-10-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1876#comment-1876 ] Marco Gaido commented on SPARK-25863: - [~Tagar] thanks for reporting this. May you p

[jira] [Updated] (SPARK-25866) Update KMeans formatVersion

2018-10-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-25866: Priority: Minor (was: Major) > Update KMeans formatVersion > --- > >

[jira] [Updated] (SPARK-25866) Update KMeans formatVersion

2018-10-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-25866: Issue Type: Bug (was: Task) > Update KMeans formatVersion > --- > >

[jira] [Created] (SPARK-25866) Update KMeans formatVersion

2018-10-29 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-25866: --- Summary: Update KMeans formatVersion Key: SPARK-25866 URL: https://issues.apache.org/jira/browse/SPARK-25866 Project: Spark Issue Type: Task Componen

[jira] [Created] (SPARK-25867) Remove KMeans computeCost

2018-10-29 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-25867: --- Summary: Remove KMeans computeCost Key: SPARK-25867 URL: https://issues.apache.org/jira/browse/SPARK-25867 Project: Spark Issue Type: Task Components

[jira] [Commented] (SPARK-25870) RandomSplit with seed gives different results depending on column order

2018-10-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667346#comment-16667346 ] Marco Gaido commented on SPARK-25870: - Why do you consider this a bug? They are 2 di

[jira] [Commented] (SPARK-25870) RandomSplit with seed gives different results depending on column order

2018-10-30 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668313#comment-16668313 ] Marco Gaido commented on SPARK-25870: - If you do some transformations (simple or com

[jira] [Commented] (SPARK-25863) java.lang.UnsupportedOperationException: empty.max at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.updateAndGetCompilationStats(CodeGenerator.scala

2018-10-30 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668438#comment-16668438 ] Marco Gaido commented on SPARK-25863: - [~Tagar] thanks. ??not sure yet as it might

[jira] [Commented] (SPARK-25441) calculate term frequency in CountVectorizer()

2018-10-30 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668654#comment-16668654 ] Marco Gaido commented on SPARK-25441: - TF has an appropriate transformer. I think th

[jira] [Commented] (SPARK-25870) RandomSplit with seed gives different results depending on column order

2018-10-30 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669037#comment-16669037 ] Marco Gaido commented on SPARK-25870: - Thanks [~deacuna]. > RandomSplit with seed g

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-11-05 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674827#comment-16674827 ] Marco Gaido commented on SPARK-24437: - Hi [~dvogelbacher], thanks for you comment an

[jira] [Commented] (SPARK-25650) Make analyzer rules used in once-policy idempotent

2018-11-05 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674955#comment-16674955 ] Marco Gaido commented on SPARK-25650: - [~maryannxue] since all the subtasks are comp

[jira] [Commented] (SPARK-25650) Make analyzer rules used in once-policy idempotent

2018-11-05 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674954#comment-16674954 ] Marco Gaido commented on SPARK-25650: - [~maryannxue] since all the subtasks are comp

[jira] [Issue Comment Deleted] (SPARK-25650) Make analyzer rules used in once-policy idempotent

2018-11-05 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-25650: Comment: was deleted (was: [~maryannxue] since all the subtasks are completed, shall we close this

[jira] [Commented] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-04 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987712#comment-16987712 ] Marco Gaido commented on SPARK-29667: - I can agree more with you [~hyukjin.kwon]. I

[jira] [Commented] (SPARK-29123) DecimalType multiplication precision loss

2019-09-19 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933128#comment-16933128 ] Marco Gaido commented on SPARK-29123: - You can set {{spark.sql.decimalOperations.all

[jira] [Commented] (SPARK-29123) DecimalType multiplication precision loss

2019-09-19 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933560#comment-16933560 ] Marco Gaido commented on SPARK-29123: - [~benny] the point here is: Spark can represe

[jira] [Commented] (SPARK-27089) Loss of precision during decimal division

2019-03-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794984#comment-16794984 ] Marco Gaido commented on SPARK-27089: - You can set: {{spark.sql.decimalOperations.al

[jira] [Updated] (SPARK-27193) CodeFormatter should format multi comment lines correctly

2019-03-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-27193: Priority: Trivial (was: Major) > CodeFormatter should format multi comment lines correctly >

[jira] [Created] (SPARK-27243) RuleExecutor throws exception when dumping time spent with no rule executed

2019-03-22 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-27243: --- Summary: RuleExecutor throws exception when dumping time spent with no rule executed Key: SPARK-27243 URL: https://issues.apache.org/jira/browse/SPARK-27243 Project: Sp

[jira] [Commented] (SPARK-27283) BigDecimal arithmetic losing precision

2019-03-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801778#comment-16801778 ] Marco Gaido commented on SPARK-27283: - [~Mats_SX] another issue which could happen u

[jira] [Updated] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

2019-03-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-27282: Priority: Major (was: Blocker) > Spark incorrect results when using UNION with GROUP BY clause >

[jira] [Commented] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

2019-03-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801930#comment-16801930 ] Marco Gaido commented on SPARK-27282: - Please do not use Blocker/Critical as they ar

[jira] [Commented] (SPARK-27283) BigDecimal arithmetic losing precision

2019-03-28 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803767#comment-16803767 ] Marco Gaido commented on SPARK-27283: - {quote} I guess I'm mostly frustrated that th

[jira] [Commented] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-02 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807780#comment-16807780 ] Marco Gaido commented on SPARK-27287: - I think the problem here is that the configur

[jira] [Commented] (SPARK-27278) Optimize GetMapValue when the map is a foldable and the key is not

2019-04-07 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16811808#comment-16811808 ] Marco Gaido commented on SPARK-27278: - [~huonw] I think the point is: in the existin

[jira] [Commented] (SPARK-26218) Throw exception on overflow for integers

2019-04-12 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816087#comment-16816087 ] Marco Gaido commented on SPARK-26218: - [~rxin] I see that. But the reason for this a

[jira] [Commented] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-23 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823876#comment-16823876 ] Marco Gaido commented on SPARK-27287: - [~dharmesh.kakadia] the point is: if you set

[jira] [Commented] (SPARK-27607) Improve performance of Row.toString()

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830933#comment-16830933 ] Marco Gaido commented on SPARK-27607: - Hi [~joshrosen], are you working on it? If no

[jira] [Commented] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830945#comment-16830945 ] Marco Gaido commented on SPARK-27612: - I am not able to reproduce... {code} _

[jira] [Commented] (SPARK-27332) Filter Pushdown duplicates expensive ScalarSubquery (discarding result)

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830947#comment-16830947 ] Marco Gaido commented on SPARK-27332: - [~dzklip] actually Spark was not using the Sc

[jira] [Commented] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831097#comment-16831097 ] Marco Gaido commented on SPARK-27612: - I don't have a python3 env, sorry... > Creat

[jira] [Resolved] (SPARK-27089) Loss of precision during decimal division

2019-05-09 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-27089. - Resolution: Information Provided > Loss of precision during decimal division > -

[jira] [Commented] (SPARK-26182) Cost increases when optimizing scalaUDF

2019-05-09 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836486#comment-16836486 ] Marco Gaido commented on SPARK-26182: - Actually you just need to mark it {{asNondete

[jira] [Resolved] (SPARK-27685) `union` doesn't promote non-nullable columns of struct to nullable

2019-05-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-27685. - Resolution: Duplicate > `union` doesn't promote non-nullable columns of struct to nullable > ---

[jira] [Commented] (SPARK-27685) `union` doesn't promote non-nullable columns of struct to nullable

2019-05-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839295#comment-16839295 ] Marco Gaido commented on SPARK-27685: - This is a duplicate of SPARK-26812. > `union

[jira] [Commented] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839297#comment-16839297 ] Marco Gaido commented on SPARK-27684: - I agree on this too. > Reduce ScalaUDF conve

[jira] [Commented] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-15 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840242#comment-16840242 ] Marco Gaido commented on SPARK-27684: - I can try and work on it, but most likely I w

[jira] [Commented] (SPARK-27761) Make UDF nondeterministic by default(?)

2019-05-20 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843914#comment-16843914 ] Marco Gaido commented on SPARK-27761: - Yes, I think this is a good idea IMHO. The be

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2019-05-25 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848097#comment-16848097 ] Marco Gaido commented on SPARK-24149: - [~Dhruve Ashar] the use case for this change,

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2019-05-30 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851896#comment-16851896 ] Marco Gaido commented on SPARK-24149: - That's true, the point is: if you want to acc

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-11-05 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675353#comment-16675353 ] Marco Gaido commented on SPARK-24437: - [~eyalfa] yes, that is the point, if there is

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-11-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679745#comment-16679745 ] Marco Gaido commented on SPARK-24437: - [~dvogelbacher] the point is: a broadcast is

[jira] [Resolved] (SPARK-25996) Agregaciones no retornan los valores correctos con rows con timestamps iguales

2018-11-10 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-25996. - Resolution: Not A Problem [~igomezraggio] check the ts of the first row. it is {{00:00:01}}, so

[jira] [Commented] (SPARK-25332) Instead of broadcast hash join ,Sort merge join has selected when restart spark-shell/spark-JDBC for hive provider

2018-11-10 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682329#comment-16682329 ] Marco Gaido commented on SPARK-25332: - [~Bjangir] please don't use "Critical" and "B

[jira] [Updated] (SPARK-25332) Instead of broadcast hash join ,Sort merge join has selected when restart spark-shell/spark-JDBC for hive provider

2018-11-10 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-25332: Priority: Major (was: Critical) > Instead of broadcast hash join ,Sort merge join has selected w

[jira] [Created] (SPARK-26003) Improve performance in SQLAppStatusListener

2018-11-10 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26003: --- Summary: Improve performance in SQLAppStatusListener Key: SPARK-26003 URL: https://issues.apache.org/jira/browse/SPARK-26003 Project: Spark Issue Type: Improv

[jira] [Created] (SPARK-26018) Support Scalar subqueries in predicate push down to datasources

2018-11-12 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26018: --- Summary: Support Scalar subqueries in predicate push down to datasources Key: SPARK-26018 URL: https://issues.apache.org/jira/browse/SPARK-26018 Project: Spark

[jira] [Commented] (SPARK-26018) Support Scalar subqueries in predicate push down to datasources

2018-11-12 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684073#comment-16684073 ] Marco Gaido commented on SPARK-26018: - I'll submit a PR for this once https://github

[jira] [Commented] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685131#comment-16685131 ] Marco Gaido commented on SPARK-26024: - I think this is the expected behavior, as Spa

[jira] [Commented] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour

2018-11-13 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685192#comment-16685192 ] Marco Gaido commented on SPARK-26024: - I am not sure about that [~JulienPeloton]. In

[jira] [Commented] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686293#comment-16686293 ] Marco Gaido commented on SPARK-26054: - I cannot reproduce this: {code} val df = Seq

[jira] [Commented] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686333#comment-16686333 ] Marco Gaido commented on SPARK-26054: - {code} val data = Seq(AA("0101", "2500.98

[jira] [Commented] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686378#comment-16686378 ] Marco Gaido commented on SPARK-26054: - Yes, sorry, I forgot to copy its definition.

[jira] [Commented] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686395#comment-16686395 ] Marco Gaido commented on SPARK-26054: - Then the affected version is 2.2.0, not 2.4.0

[jira] [Updated] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-26054: Component/s: (was: Spark Core) SQL > Creating a computed column applying the

[jira] [Resolved] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-26054. - Resolution: Cannot Reproduce > Creating a computed column applying the spark sql rounding on a c

[jira] [Updated] (SPARK-26054) Creating a computed column applying the spark sql rounding on a column of type decimal affects the orginal column as well.

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-26054: Affects Version/s: (was: 2.4.0) 2.2.0 > Creating a computed column appl

[jira] [Commented] (SPARK-26041) catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686828#comment-16686828 ] Marco Gaido commented on SPARK-26041: - I think this may be a duplicate of SPARK-2605

[jira] [Commented] (SPARK-26041) catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686857#comment-16686857 ] Marco Gaido commented on SPARK-26041: - Then it'd help if you could provide a reprodu

[jira] [Commented] (SPARK-26041) catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute

2018-11-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686870#comment-16686870 ] Marco Gaido commented on SPARK-26041: - No, it is not, for 2.3 we would need a dedica

[jira] [Resolved] (SPARK-26018) Support Scalar subqueries in predicate push down to datasources

2018-11-15 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-26018. - Resolution: Won't Fix This may be very hard to do as we now add filters to datasources from logi

[jira] [Commented] (SPARK-26041) catalyst cuts out some columns from dataframes: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute

2018-11-15 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687951#comment-16687951 ] Marco Gaido commented on SPARK-26041: - [~Tagar] I don't have you table definitions s

[jira] [Commented] (SPARK-26063) CatalystDataToAvro gives "UnresolvedException: Invalid call to dataType on unresolved object" when requested for numberedTreeString

2018-11-15 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687971#comment-16687971 ] Marco Gaido commented on SPARK-26063: - I think this was fixed in SPARK-25883. But we

[jira] [Commented] (SPARK-26045) Error in the spark 2.4 release package with the spark-avro_2.11 depdency

2018-11-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689217#comment-16689217 ] Marco Gaido commented on SPARK-26045: - [~o.garcia] can you please create a PR for th

[jira] [Commented] (SPARK-26078) WHERE .. IN fails to filter rows when used in combination with UNION

2018-11-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689262#comment-16689262 ] Marco Gaido commented on SPARK-26078: - I'll investigate this immediately, thanks [~c

[jira] [Commented] (SPARK-25959) Difference in featureImportances results on computed vs saved models

2018-11-20 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692861#comment-16692861 ] Marco Gaido commented on SPARK-25959: - [~srowen] what do you think about backporting

[jira] [Updated] (SPARK-26127) Remove deprecated setImpurity from tree regression and classification models

2018-11-20 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-26127: Summary: Remove deprecated setImpurity from tree regression and classification models (was: Remov

[jira] [Created] (SPARK-26127) Remove deprecated setImpurity from GBTClassificationModel, DecisionTreeRegressionModel, GBTRegressionModel, RandomForestRegressionModel

2018-11-20 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26127: --- Summary: Remove deprecated setImpurity from GBTClassificationModel, DecisionTreeRegressionModel, GBTRegressionModel, RandomForestRegressionModel Key: SPARK-26127 URL: https://issue

[jira] [Updated] (SPARK-26127) Remove deprecated setters from tree regression and classification models

2018-11-20 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-26127: Summary: Remove deprecated setters from tree regression and classification models (was: Remove de

[jira] [Updated] (SPARK-26127) Remove deprecated setters from tree regression and classification models

2018-11-20 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-26127: Description: Many {{set***}} methods are present for the models of regression and classification t

[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

2018-11-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702889#comment-16702889 ] Marco Gaido commented on SPARK-24498: - +1 for closing this. > Add JDK compiler for

[jira] [Commented] (SPARK-26214) Add "broadcast" method to DataFrame

2018-11-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702994#comment-16702994 ] Marco Gaido commented on SPARK-26214: - You can just use the {{broadcast}} function f

[jira] [Commented] (SPARK-26215) define reserved keywords after SQL standard

2018-11-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703042#comment-16703042 ] Marco Gaido commented on SPARK-26215: - [~cloud_fan] thanks for pinging me. I agree o

[jira] [Created] (SPARK-26217) Compliance to SQL standard (SQL:2011)

2018-11-29 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26217: --- Summary: Compliance to SQL standard (SQL:2011) Key: SPARK-26217 URL: https://issues.apache.org/jira/browse/SPARK-26217 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-23179) Support option to throw exception if overflow occurs

2018-11-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-23179: Issue Type: Sub-task (was: Improvement) Parent: SPARK-26217 > Support option to throw exc

[jira] [Updated] (SPARK-26215) define reserved keywords after SQL standard

2018-11-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-26215: Issue Type: Sub-task (was: Improvement) Parent: SPARK-26217 > define reserved keywords af

[jira] [Created] (SPARK-26218) Throw exception on overflow for integers

2018-11-29 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26218: --- Summary: Throw exception on overflow for integers Key: SPARK-26218 URL: https://issues.apache.org/jira/browse/SPARK-26218 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-26214) Add "broadcast" method to DataFrame

2018-11-30 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704958#comment-16704958 ] Marco Gaido commented on SPARK-26214: - I don't think it is really the same. I don't

[jira] [Commented] (SPARK-26242) Leading slash breaks proxying

2018-12-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705916#comment-16705916 ] Marco Gaido commented on SPARK-26242: - You can set {{spark.ui.proxyBase}} or the pro

[jira] [Commented] (SPARK-26242) Leading slash breaks proxying

2018-12-02 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706157#comment-16706157 ] Marco Gaido commented on SPARK-26242: - Let me close this. Please reopen only if you

[jira] [Resolved] (SPARK-26242) Leading slash breaks proxying

2018-12-02 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-26242. - Resolution: Not A Problem > Leading slash breaks proxying > - > >

[jira] [Commented] (SPARK-26233) Incorrect decimal value with java beans and first/last/max... functions

2018-12-03 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707009#comment-16707009 ] Marco Gaido commented on SPARK-26233: - I think this is related to SPARK-24957. The p

[jira] [Commented] (SPARK-26233) Incorrect decimal value with java beans and first/last/max... functions

2018-12-03 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707729#comment-16707729 ] Marco Gaido commented on SPARK-26233: - [~dongjoon] I think so. SPARK-24957 was a lon

<    1   2   3   4   5   6   7   >