[jira] [Commented] (SPARK-22865) Publish Official Apache Spark Docker images

2018-03-22 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409351#comment-16409351 ] Andrew Korzhuev commented on SPARK-22865: - What is the plan on resolving this issue? I see that

[jira] [Created] (SPARK-23668) Support for imagePullSecrets k8s option

2018-03-13 Thread Andrew Korzhuev (JIRA)
Andrew Korzhuev created SPARK-23668: --- Summary: Support for imagePullSecrets k8s option Key: SPARK-23668 URL: https://issues.apache.org/jira/browse/SPARK-23668 Project: Spark Issue Type:

[jira] [Updated] (SPARK-23449) Extra java options lose order in Docker context

2018-02-16 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Korzhuev updated SPARK-23449: Description: `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` when

[jira] [Created] (SPARK-23449) Extra java options lose order in Docker context

2018-02-16 Thread Andrew Korzhuev (JIRA)
Andrew Korzhuev created SPARK-23449: --- Summary: Extra java options lose order in Docker context Key: SPARK-23449 URL: https://issues.apache.org/jira/browse/SPARK-23449 Project: Spark Issue

[jira] [Commented] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

2018-01-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347480#comment-16347480 ] Andrew Ash commented on SPARK-23274: Many thanks for the fast fix [~smilegator]! >

[jira] [Commented] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

2018-01-30 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345539#comment-16345539 ] Andrew Ash commented on SPARK-23274: Suspect this regression was introduced by

[jira] [Commented] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

2018-01-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330943#comment-16330943 ] Andrew Ash commented on SPARK-22982: [~joshrosen] do you have some example stacktraces of what this

[jira] [Created] (SPARK-23133) Spark options are not passed to the Executor in Docker context

2018-01-17 Thread Andrew Korzhuev (JIRA)
Andrew Korzhuev created SPARK-23133: --- Summary: Spark options are not passed to the Executor in Docker context Key: SPARK-23133 URL: https://issues.apache.org/jira/browse/SPARK-23133 Project: Spark

[jira] [Commented] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281354#comment-16281354 ] Andrew Ash commented on SPARK-22725: Demonstration of difference between {{.map}} on List vs Stream:

[jira] [Created] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22725: -- Summary: df.select on a Stream is broken, vs a List Key: SPARK-22725 URL: https://issues.apache.org/jira/browse/SPARK-22725 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-29 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-29 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Commented] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270132#comment-16270132 ] Andrew Duffy commented on SPARK-22641: -- Query plan with the literal: {code} == Parsed Logical Plan

[jira] [Commented] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270128#comment-16270128 ] Andrew Duffy commented on SPARK-22641: -- So it seems this is only a problem when using literal

[jira] [Comment Edited] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270128#comment-16270128 ] Andrew Duffy edited comment on SPARK-22641 at 11/29/17 4:44 AM: So it

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Created] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
Andrew Duffy created SPARK-22641: Summary: Pyspark UDF relying on column added with withColumn after distinct Key: SPARK-22641 URL: https://issues.apache.org/jira/browse/SPARK-22641 Project: Spark

[jira] [Commented] (SPARK-22541) Dataframes: applying multiple filters one after another using udfs and accumulators results in faulty accumulators

2017-11-27 Thread Andrew Snare (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267567#comment-16267567 ] Andrew Snare commented on SPARK-22541: -- Although this ship has sailed, a search brought me here

[jira] [Created] (SPARK-22555) Possibly incorrect scaling of L2 regularization strength in LinearRegression

2017-11-19 Thread Andrew Crosby (JIRA)
Andrew Crosby created SPARK-22555: - Summary: Possibly incorrect scaling of L2 regularization strength in LinearRegression Key: SPARK-22555 URL: https://issues.apache.org/jira/browse/SPARK-22555

[jira] [Commented] (SPARK-22479) SaveIntoDataSourceCommand logs jdbc credentials

2017-11-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246328#comment-16246328 ] Andrew Ash commented on SPARK-22479: Completely agree that credentials shouldn't be in the toString

[jira] [Created] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

2017-11-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22470: -- Summary: Doc that functions.hash is also used internally for shuffle and bucketing Key: SPARK-22470 URL: https://issues.apache.org/jira/browse/SPARK-22470 Project: Spark

[jira] [Commented] (SPARK-22042) ReorderJoinPredicates can break when child's partitioning is not decided

2017-10-26 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220734#comment-16220734 ] Andrew Ash commented on SPARK-22042: Hi I'm seeing this problem as well, thanks for investigating and

[jira] [Commented] (SPARK-21991) [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

2017-10-25 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219139#comment-16219139 ] Andrew Ash commented on SPARK-21991: Thanks for the contribution to Spark [~nivox]! I'll be testing

[jira] [Commented] (SPARK-21991) [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

2017-10-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216529#comment-16216529 ] Andrew Ash commented on SPARK-21991: Thanks for debugging and diagnosing this [~nivox]! I'm seeing

[jira] [Commented] (SPARK-22204) Explain output for SQL with commands shows no optimization

2017-10-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208676#comment-16208676 ] Andrew Ash commented on SPARK-22204: One way to work around this issue could be by getting the child

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202795#comment-16202795 ] Andrew Ash commented on SPARK-22269: [~sowen] you closed this as a duplicate. What issue is it a

[jira] [Commented] (SPARK-22268) Fix java style errors

2017-10-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202793#comment-16202793 ] Andrew Ash commented on SPARK-22268: Any time {{./dev/run-tests}} is failing I consider that a bug.

[jira] [Created] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22269: -- Summary: Java style checks should be run in Jenkins Key: SPARK-22269 URL: https://issues.apache.org/jira/browse/SPARK-22269 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-22268) Fix java style errors

2017-10-12 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22268: -- Summary: Fix java style errors Key: SPARK-22268 URL: https://issues.apache.org/jira/browse/SPARK-22268 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-18359) Let user specify locale in CSV parsing

2017-10-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200831#comment-16200831 ] Andrew Ash commented on SPARK-18359: I agree with Sean -- using the submitting JVM's locale is

[jira] [Commented] (SPARK-20055) Documentation for CSV datasets in SQL programming guide

2017-10-05 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193355#comment-16193355 ] Andrew Ash commented on SPARK-20055: What I would find most useful is a list of available options and

[jira] [Created] (SPARK-22204) Explain output for SQL with commands shows no optimization

2017-10-04 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22204: -- Summary: Explain output for SQL with commands shows no optimization Key: SPARK-22204 URL: https://issues.apache.org/jira/browse/SPARK-22204 Project: Spark Issue

[jira] [Reopened] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2017-09-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash reopened SPARK-18016: // reopening issue One PR addressing this bug has been merged --

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-09-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178459#comment-16178459 ] Andrew Ash commented on SPARK-19700: There was a thread on the dev list recently about Apache Aurora:

[jira] [Created] (SPARK-22112) Add missing method to pyspark api: spark.read.csv(Dataset)

2017-09-24 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22112: -- Summary: Add missing method to pyspark api: spark.read.csv(Dataset) Key: SPARK-22112 URL: https://issues.apache.org/jira/browse/SPARK-22112 Project: Spark

[jira] [Created] (SPARK-21962) Distributed Tracing in Spark

2017-09-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21962: -- Summary: Distributed Tracing in Spark Key: SPARK-21962 URL: https://issues.apache.org/jira/browse/SPARK-21962 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-21953) Show both memory and disk bytes spilled if either is present

2017-09-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21953: -- Summary: Show both memory and disk bytes spilled if either is present Key: SPARK-21953 URL: https://issues.apache.org/jira/browse/SPARK-21953 Project: Spark

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2017-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156622#comment-16156622 ] Andrew Ash commented on SPARK-12449: [~velvia] I'm not involved with the CatalystSource or SAP

[jira] [Created] (SPARK-21941) Stop storing unused attemptId in SQLTaskMetrics

2017-09-07 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21941: -- Summary: Stop storing unused attemptId in SQLTaskMetrics Key: SPARK-21941 URL: https://issues.apache.org/jira/browse/SPARK-21941 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21807) The getAliasedConstraints function in LogicalPlan will take a long time when number of expressions is greater than 100

2017-08-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149600#comment-16149600 ] Andrew Ash commented on SPARK-21807: For reference, here's a stacktrace I'm seeing on a cluster

[jira] [Commented] (SPARK-21875) Jenkins passes Java code that violates ./dev/lint-java

2017-08-30 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147976#comment-16147976 ] Andrew Ash commented on SPARK-21875: I'd be interested in more details on why it can't be run in the

[jira] [Created] (SPARK-21875) Jenkins passes Java code that violates ./dev/lint-java

2017-08-30 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21875: -- Summary: Jenkins passes Java code that violates ./dev/lint-java Key: SPARK-21875 URL: https://issues.apache.org/jira/browse/SPARK-21875 Project: Spark Issue

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138509#comment-16138509 ] Andrew Ash commented on SPARK-15689: Can the authors of this document add a section contrasting the

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2017-08-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138501#comment-16138501 ] Andrew Ash commented on SPARK-12449: Relevant slides:

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-08-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136371#comment-16136371 ] Andrew Ash commented on SPARK-19552: I didn't see anything other than the issue you just commented on

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-08-20 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134657#comment-16134657 ] Andrew Ash commented on SPARK-19552: Heads up the next time someone attempts this: Upgrading to

[jira] [Created] (SPARK-21768) spark.csv.read Empty String Parsed as NULL when nullValue is Set

2017-08-17 Thread Andrew Gross (JIRA)
Andrew Gross created SPARK-21768: Summary: spark.csv.read Empty String Parsed as NULL when nullValue is Set Key: SPARK-21768 URL: https://issues.apache.org/jira/browse/SPARK-21768 Project: Spark

[jira] [Updated] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

2017-08-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21757: --- Description: At the following two places if the {{e.reason}} value contains a single quote

[jira] [Created] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

2017-08-16 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21757: -- Summary: Jobs page fails to load when executor removed event's reason contains single quote Key: SPARK-21757 URL: https://issues.apache.org/jira/browse/SPARK-21757

[jira] [Commented] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-08-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122651#comment-16122651 ] Andrew Ash commented on SPARK-21564: [~irashid] a possible fix could look roughly like this:

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-08-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122549#comment-16122549 ] Andrew Ash commented on SPARK-21563: Thanks for the thoughts [~irashid] -- I submitted a PR

[jira] [Commented] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

2017-08-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120643#comment-16120643 ] Andrew Ash commented on SPARK-19116: Ah yes, for files it seems like Spark currently uses size of the

[jira] [Closed] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

2017-08-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-19116. -- Resolution: Not A Problem > LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file >

[jira] [Commented] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

2017-08-04 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114751#comment-16114751 ] Andrew Ash commented on SPARK-19116: [~shea.parkes] does this answer your question? >

[jira] [Created] (SPARK-21628) Explicitly specify Java version in maven compiler plugin so IntelliJ imports project correctly

2017-08-03 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21628: -- Summary: Explicitly specify Java version in maven compiler plugin so IntelliJ imports project correctly Key: SPARK-21628 URL: https://issues.apache.org/jira/browse/SPARK-21628

[jira] [Commented] (SPARK-21034) Filter not getting pushed down the groupBy clause when first() or last() aggregate function is used

2017-08-02 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111454#comment-16111454 ] Andrew Ray commented on SPARK-21034: Yes a=1 is the filter to be pushed down. It is not pushed

[jira] [Commented] (SPARK-21034) Filter not getting pushed down the groupBy clause when first() or last() aggregate function is used

2017-08-02 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1631#comment-1631 ] Andrew Ray commented on SPARK-21034: {{first}} is not a deterministic function and thus filters are

[jira] [Commented] (SPARK-21110) Structs should be usable in inequality filters

2017-08-02 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111093#comment-16111093 ] Andrew Ray commented on SPARK-21110: https://github.com/apache/spark/pull/18818 > Structs should be

[jira] [Commented] (SPARK-21110) Structs should be usable in inequality filters

2017-08-01 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109734#comment-16109734 ] Andrew Ray commented on SPARK-21110: I'm working on this > Structs should be usable in inequality

[jira] [Commented] (SPARK-21330) Bad partitioning does not allow to read a JDBC table with extreme values on the partition column

2017-08-01 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109568#comment-16109568 ] Andrew Ray commented on SPARK-21330: https://github.com/apache/spark/pull/18800 > Bad partitioning

[jira] [Commented] (SPARK-20433) Update jackson-databind to 2.6.7.1

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108117#comment-16108117 ] Andrew Ash commented on SPARK-20433: Sorry about not updating the ticket description -- the 2.6.7.1

[jira] [Updated] (SPARK-20433) Update jackson-databind to 2.6.7.1

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Description: There was a security vulnerability recently reported to the upstream jackson-databind

[jira] [Updated] (SPARK-20433) Update jackson-databind to 2.6.7.1

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Summary: Update jackson-databind to 2.6.7.1 (was: Update jackson-databind to 2.6.7) > Update

[jira] [Reopened] (SPARK-20433) Update jackson-databind to 2.6.7

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash reopened SPARK-20433: > Update jackson-databind to 2.6.7 > > > Key:

[jira] [Commented] (SPARK-21565) aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp

2017-07-31 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108006#comment-16108006 ] Andrew Ray commented on SPARK-21565: No nothing like the limitations of microbatches. The window can

[jira] [Commented] (SPARK-21565) aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp

2017-07-31 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107933#comment-16107933 ] Andrew Ray commented on SPARK-21565: I believe you need to use a window to group by your event time.

[jira] [Updated] (SPARK-21584) Update R method for summary to call new implementation

2017-07-31 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ray updated SPARK-21584: --- Component/s: SQL > Update R method for summary to call new implementation >

[jira] [Created] (SPARK-21584) Update R method for summary to call new implementation

2017-07-31 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21584: -- Summary: Update R method for summary to call new implementation Key: SPARK-21584 URL: https://issues.apache.org/jira/browse/SPARK-21584 Project: Spark Issue

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105951#comment-16105951 ] Andrew Ash commented on SPARK-20433: As I wrote in that PR, it's 2.6.7.1 of jackson-databind that has

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105772#comment-16105772 ] Andrew Ash commented on SPARK-21563: And for reference, I added this additional logging to assist in

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105751#comment-16105751 ] Andrew Ash commented on SPARK-20433: Here's the patch I put in my fork of Spark:

[jira] [Created] (SPARK-21566) Python method for summary

2017-07-28 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21566: -- Summary: Python method for summary Key: SPARK-21566 URL: https://issues.apache.org/jira/browse/SPARK-21566 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21564: --- Description: cc [~robert3005] I was seeing an issue where Spark was throwing this exception:

[jira] [Updated] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21563: --- Description: cc [~robert3005] I was seeing this exception during some running Spark jobs:

[jira] [Created] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21564: -- Summary: TaskDescription decoding failure should fail the task Key: SPARK-21564 URL: https://issues.apache.org/jira/browse/SPARK-21564 Project: Spark Issue

[jira] [Created] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21563: -- Summary: Race condition when serializing TaskDescriptions and adding jars Key: SPARK-21563 URL: https://issues.apache.org/jira/browse/SPARK-21563 Project: Spark

[jira] [Commented] (SPARK-14887) Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits

2017-07-25 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099584#comment-16099584 ] Andrew Ash commented on SPARK-14887: [~fang fang chen] have you seen this in the latest version of

[jira] [Commented] (SPARK-21289) Text and CSV formats do not support custom end-of-line delimiters

2017-07-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081136#comment-16081136 ] Andrew Ash commented on SPARK-21289: Looks like this will fix SPARK-17227 also > Text and CSV

[jira] [Closed] (SPARK-15226) CSV file data-line with newline at first line load error

2017-07-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-15226. -- Resolution: Fixed Fix Version/s: 2.2.0 Fixed by Fixed by

[jira] [Comment Edited] (SPARK-15226) CSV file data-line with newline at first line load error

2017-07-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081154#comment-16081154 ] Andrew Ash edited comment on SPARK-15226 at 7/10/17 9:07 PM: - Fixed by

[jira] [Updated] (SPARK-21100) Add summary method as alternative to describe that gives quartiles similar to Pandas

2017-07-05 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ray updated SPARK-21100: --- Summary: Add summary method as alternative to describe that gives quartiles similar to Pandas (was:

[jira] [Commented] (SPARK-21184) QuantileSummaries implementation is wrong and QuantileSummariesSuite fails with larger n

2017-06-28 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067167#comment-16067167 ] Andrew Ray commented on SPARK-21184: Also the lookup queries are just wrong {code} scala> Seq(1,

[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

2017-06-27 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065177#comment-16065177 ] Andrew Duffy edited comment on SPARK-21218 at 6/27/17 5:39 PM: --- Curious, I

[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

2017-06-27 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065177#comment-16065177 ] Andrew Duffy commented on SPARK-21218: -- Curious, I wonder what the previous benchmarks were lacking.

[jira] [Created] (SPARK-21220) Use outputPartitioning's bucketing if possible on write

2017-06-26 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21220: -- Summary: Use outputPartitioning's bucketing if possible on write Key: SPARK-21220 URL: https://issues.apache.org/jira/browse/SPARK-21220 Project: Spark Issue

[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

2017-06-26 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063981#comment-16063981 ] Andrew Duffy commented on SPARK-21218: -- Good catch, looks like a dupe. [~hyukjin.kwon] did profiling

[jira] [Resolved] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq

2017-06-26 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy resolved SPARK-17091. -- Resolution: Won't Fix Should've closed this last year, but at the time based on Hyukjin Kwon's

[jira] [Created] (SPARK-21184) QuantileSummaries implementation is wrong and QuantileSummariesSuite fails with larger n

2017-06-22 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21184: -- Summary: QuantileSummaries implementation is wrong and QuantileSummariesSuite fails with larger n Key: SPARK-21184 URL: https://issues.apache.org/jira/browse/SPARK-21184

[jira] [Comment Edited] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058930#comment-16058930 ] Andrew Ash edited comment on SPARK-19700 at 6/22/17 7:47 AM: - Public

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058930#comment-16058930 ] Andrew Ash commented on SPARK-19700: Public implementation that's been around a while: Two Sigma's

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058917#comment-16058917 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: Facebook's in-house

[jira] [Created] (SPARK-21100) describe should give quartiles similar to Pandas

2017-06-14 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21100: -- Summary: describe should give quartiles similar to Pandas Key: SPARK-21100 URL: https://issues.apache.org/jira/browse/SPARK-21100 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-20839) Incorrect Dynamic PageRank calculation

2017-06-14 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ray resolved SPARK-20839. Resolution: Not A Problem > Incorrect Dynamic PageRank calculation >

[jira] [Commented] (SPARK-20839) Incorrect Dynamic PageRank calculation

2017-06-14 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049214#comment-16049214 ] Andrew Ray commented on SPARK-20839: 1 & 2 work together to do the algorithm properly with an active

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041803#comment-16041803 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: Nomad by [~barnardb] at

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-02 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035171#comment-16035171 ] Andrew Ash commented on SPARK-20952: For the localProperties on SparkContext it does 2 things I can

[jira] [Created] (SPARK-20815) NullPointerException in RPackageUtils#checkManifestForR

2017-05-19 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-20815: -- Summary: NullPointerException in RPackageUtils#checkManifestForR Key: SPARK-20815 URL: https://issues.apache.org/jira/browse/SPARK-20815 Project: Spark Issue

<    1   2   3   4   5   6   7   8   9   10   >