[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541265#comment-16541265 ] Andrew Ash commented on SPARK-21962: Note that HTrace is now being removed

Re: [VOTE] Spark 2.3.0 (RC2)

was resolved > yesterday and tests have been quite healthy throughout this week and the > last. I'll cut the new RC as soon as the remaining blocker (SPARK-23202 > <https://issues.apache.org/jira/browse/SPARK-23202>) is resolved. > > > On 30 January 2018 at 10:12, Andr

[jira] [Commented] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347480#comment-16347480 ] Andrew Ash commented on SPARK-23274: Many thanks for the fast fix [~smileg

Re: [VOTE] Spark 2.3.0 (RC2)

I'd like to nominate SPARK-23274 as a potential blocker for the 2.3.0 release as well, due to being a regression from 2.2.0. The ticket has a simple repro included, showing a query that works in prior releases but now fails with an exception in t

[jira] [Commented] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345539#comment-16345539 ] Andrew Ash commented on SPARK-23274: Suspect this regression was introduce

[jira] [Commented] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

[ https://issues.apache.org/jira/browse/SPARK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16330943#comment-16330943 ] Andrew Ash commented on SPARK-22982: [~joshrosen] do you have some exa

Re: Kubernetes: why use init containers?

+1 on the first release being marked experimental. Many major features coming into Spark in the past have gone through a stabilization process On Fri, Jan 12, 2018 at 1:18 PM, Marcelo Vanzin wrote: > BTW I most probably will not have time to get back to this at any time > soon, so if anyone is

Re: Kubernetes: why use init containers?

It seems we have two standard practices for resource distribution in place here: - the Spark way is that the application (Spark) distributes the resources *during* app execution, and does this by exposing files/jars on an http server on the driver (or pre-staged elsewhere), and executors downloadi

Re: Palantir replease under org.apache.spark?

That source repo is at https://github.com/palantir/spark/ with artifacts published to Palantir's bintray at https://palantir.bintray.com/releases/org/apache/spark/ If you're seeing any of them in Maven Central please flag, as that's a mistake! Andrew On Tue, Jan 9, 2018 at 10:10 AM, Sean Owen w

Re: Palantir replease under org.apache.spark?

That source repo is at https://github.com/palantir/spark/ with artifacts published to Palantir's bintray at https://palantir.bintray.com/releases/org/apache/spark/ If you're seeing any of them in Maven Central please flag, as that's a mistake! Andrew On Tue, Jan 9, 2018 at 10:10 AM, Sean Owen w

[jira] [Commented] (SPARK-22725) df.select on a Stream is broken, vs a List

[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281354#comment-16281354 ] Andrew Ash commented on SPARK-22725: Demonstration of difference between {{.map}

[jira] [Created] (SPARK-22725) df.select on a Stream is broken, vs a List

Andrew Ash created SPARK-22725: -- Summary: df.select on a Stream is broken, vs a List Key: SPARK-22725 URL: https://issues.apache.org/jira/browse/SPARK-22725 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22479) SaveIntoDataSourceCommand logs jdbc credentials

[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246328#comment-16246328 ] Andrew Ash commented on SPARK-22479: Completely agree that credentials shouldn&

[jira] [Created] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

Andrew Ash created SPARK-22470: -- Summary: Doc that functions.hash is also used internally for shuffle and bucketing Key: SPARK-22470 URL: https://issues.apache.org/jira/browse/SPARK-22470 Project: Spark

[jira] [Commented] (SPARK-22042) ReorderJoinPredicates can break when child's partitioning is not decided

[ https://issues.apache.org/jira/browse/SPARK-22042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220734#comment-16220734 ] Andrew Ash commented on SPARK-22042: Hi I'm seeing this problem as well, t

[jira] [Commented] (SPARK-21991) [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

[ https://issues.apache.org/jira/browse/SPARK-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219139#comment-16219139 ] Andrew Ash commented on SPARK-21991: Thanks for the contribution to Spark [~n

[jira] [Commented] (SPARK-21991) [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

[ https://issues.apache.org/jira/browse/SPARK-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216529#comment-16216529 ] Andrew Ash commented on SPARK-21991: Thanks for debugging and diagnosing

[jira] [Commented] (SPARK-22204) Explain output for SQL with commands shows no optimization

[ https://issues.apache.org/jira/browse/SPARK-22204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208676#comment-16208676 ] Andrew Ash commented on SPARK-22204: One way to work around this issue could b

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202795#comment-16202795 ] Andrew Ash commented on SPARK-22269: [~sowen] you closed this as a duplicate.

[jira] [Commented] (SPARK-22268) Fix java style errors

[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202793#comment-16202793 ] Andrew Ash commented on SPARK-22268: Any time {{./dev/run-tests}} is faili

[jira] [Created] (SPARK-22269) Java style checks should be run in Jenkins

Andrew Ash created SPARK-22269: -- Summary: Java style checks should be run in Jenkins Key: SPARK-22269 URL: https://issues.apache.org/jira/browse/SPARK-22269 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-22268) Fix java style errors

Andrew Ash created SPARK-22268: -- Summary: Fix java style errors Key: SPARK-22268 URL: https://issues.apache.org/jira/browse/SPARK-22268 Project: Spark Issue Type: Bug Components

[jira] [Commented] (SPARK-18359) Let user specify locale in CSV parsing

[ https://issues.apache.org/jira/browse/SPARK-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200831#comment-16200831 ] Andrew Ash commented on SPARK-18359: I agree with Sean -- using the submitting J

[jira] [Commented] (SPARK-20055) Documentation for CSV datasets in SQL programming guide

[ https://issues.apache.org/jira/browse/SPARK-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193355#comment-16193355 ] Andrew Ash commented on SPARK-20055: What I would find most useful is a lis

[jira] [Created] (SPARK-22204) Explain output for SQL with commands shows no optimization

Andrew Ash created SPARK-22204: -- Summary: Explain output for SQL with commands shows no optimization Key: SPARK-22204 URL: https://issues.apache.org/jira/browse/SPARK-22204 Project: Spark Issue

[jira] [Reopened] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash reopened SPARK-18016: // reopening issue One PR addressing this bug has been merged -- https://github.com/apache/spark

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178459#comment-16178459 ] Andrew Ash commented on SPARK-19700: There was a thread on the dev list rece

[jira] [Created] (SPARK-22112) Add missing method to pyspark api: spark.read.csv(Dataset)

Andrew Ash created SPARK-22112: -- Summary: Add missing method to pyspark api: spark.read.csv(Dataset) Key: SPARK-22112 URL: https://issues.apache.org/jira/browse/SPARK-22112 Project: Spark

[jira] [Created] (SPARK-21962) Distributed Tracing in Spark

Andrew Ash created SPARK-21962: -- Summary: Distributed Tracing in Spark Key: SPARK-21962 URL: https://issues.apache.org/jira/browse/SPARK-21962 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-21953) Show both memory and disk bytes spilled if either is present

Andrew Ash created SPARK-21953: -- Summary: Show both memory and disk bytes spilled if either is present Key: SPARK-21953 URL: https://issues.apache.org/jira/browse/SPARK-21953 Project: Spark

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

+0 (non-binding) I think there are benefits to unifying all the Spark-internal datasources into a common public API for sure. It will serve as a forcing function to ensure that those internal datasources aren't advantaged vs datasources developed externally as plugins to Spark, and that all Spark

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156622#comment-16156622 ] Andrew Ash commented on SPARK-12449: [~velvia] I'm not involved

[jira] [Created] (SPARK-21941) Stop storing unused attemptId in SQLTaskMetrics

Andrew Ash created SPARK-21941: -- Summary: Stop storing unused attemptId in SQLTaskMetrics Key: SPARK-21941 URL: https://issues.apache.org/jira/browse/SPARK-21941 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21807) The getAliasedConstraints function in LogicalPlan will take a long time when number of expressions is greater than 100

[ https://issues.apache.org/jira/browse/SPARK-21807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149600#comment-16149600 ] Andrew Ash commented on SPARK-21807: For reference, here's a stacktrace I

[jira] [Commented] (SPARK-21875) Jenkins passes Java code that violates ./dev/lint-java

[ https://issues.apache.org/jira/browse/SPARK-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147976#comment-16147976 ] Andrew Ash commented on SPARK-21875: I'd be interested in more details

[jira] [Created] (SPARK-21875) Jenkins passes Java code that violates ./dev/lint-java

Andrew Ash created SPARK-21875: -- Summary: Jenkins passes Java code that violates ./dev/lint-java Key: SPARK-21875 URL: https://issues.apache.org/jira/browse/SPARK-21875 Project: Spark Issue

[jira] [Commented] (SPARK-15689) Data source API v2

[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138509#comment-16138509 ] Andrew Ash commented on SPARK-15689: Can the authors of this document add a sec

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138501#comment-16138501 ] Andrew Ash commented on SPARK-12449: Relevant slides: https://www.slideshare

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136371#comment-16136371 ] Andrew Ash commented on SPARK-19552: I didn't see anything other than the

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134657#comment-16134657 ] Andrew Ash commented on SPARK-19552: Heads up the next time someone attempts

[jira] [Updated] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

[ https://issues.apache.org/jira/browse/SPARK-21757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21757: --- Description: At the following two places if the {{e.reason}} value contains a single quote

[jira] [Created] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

Andrew Ash created SPARK-21757: -- Summary: Jobs page fails to load when executor removed event's reason contains single quote Key: SPARK-21757 URL: https://issues.apache.org/jira/browse/SPARK-

Re: SPIP: Spark on Kubernetes

+1 (non-binding) We're moving large amounts of infrastructure from a combination of open source and homegrown cluster management systems to unify on Kubernetes and want to bring Spark workloads along with us. On Tue, Aug 15, 2017 at 2:29 PM, liyinan926 wrote: > +1 (non-binding) > > > > -- > Vie

Re: Use Apache ORC in Apache Spark 2.3

ORC > codes. > > > > And, Spark without `-Phive` can ORC like Parquet. > > > > This is one milestone for `Feature parity for ORC with Parquet > (SPARK-20901)`. > > > > Bests, > > Dongjoon > > > > *From: *Reynold Xin > *Date:

[jira] [Commented] (SPARK-21564) TaskDescription decoding failure should fail the task

[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122651#comment-16122651 ] Andrew Ash commented on SPARK-21564: [~irashid] a possible fix could look rou

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122549#comment-16122549 ] Andrew Ash commented on SPARK-21563: Thanks for the thoughts [~irashid]

Re: Use Apache ORC in Apache Spark 2.3

I would support moving ORC from sql/hive -> sql/core because it brings me one step closer to eliminating Hive from my Spark distribution by removing -Phive at build time. On Thu, Aug 10, 2017 at 9:48 AM, Dong Joon Hyun wrote: > Thank you again for coming and reviewing this PR. > > > > So far, we

[jira] [Commented] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120643#comment-16120643 ] Andrew Ash commented on SPARK-19116: Ah yes, for files it seems like Spark curre

[jira] [Closed] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-19116. -- Resolution: Not A Problem > LogicalPlan.statistics.sizeInBytes wrong for trivial parquet f

[jira] [Commented] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114751#comment-16114751 ] Andrew Ash commented on SPARK-19116: [~shea.parkes] does this answer your ques

[jira] [Commented] (SPARK-20433) Update jackson-databind to 2.6.7.1

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108117#comment-16108117 ] Andrew Ash commented on SPARK-20433: Sorry about not updating the ticket descrip

[jira] [Updated] (SPARK-20433) Update jackson-databind to 2.6.7.1

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Description: There was a security vulnerability recently reported to the upstream jackson-databind

[jira] [Updated] (SPARK-20433) Update jackson-databind to 2.6.7.1

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Summary: Update jackson-databind to 2.6.7.1 (was: Update jackson-databind to 2.6.7) > Upd

[jira] [Reopened] (SPARK-20433) Update jackson-databind to 2.6.7

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash reopened SPARK-20433: > Update jackson-databind to 2.6.7 > > >

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105951#comment-16105951 ] Andrew Ash commented on SPARK-20433: As I wrote in that PR, it's 2.6.7.1 o

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105772#comment-16105772 ] Andrew Ash commented on SPARK-21563: And for reference, I added this additi

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105751#comment-16105751 ] Andrew Ash commented on SPARK-20433: Here's the patch I put in my fork

[jira] [Updated] (SPARK-21564) TaskDescription decoding failure should fail the task

[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21564: --- Description: cc [~robert3005] I was seeing an issue where Spark was throwing this exception

[jira] [Updated] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21563: --- Description: cc [~robert3005] I was seeing this exception during some running Spark jobs

[jira] [Created] (SPARK-21564) TaskDescription decoding failure should fail the task

Andrew Ash created SPARK-21564: -- Summary: TaskDescription decoding failure should fail the task Key: SPARK-21564 URL: https://issues.apache.org/jira/browse/SPARK-21564 Project: Spark Issue Type

[jira] [Created] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

Andrew Ash created SPARK-21563: -- Summary: Race condition when serializing TaskDescriptions and adding jars Key: SPARK-21563 URL: https://issues.apache.org/jira/browse/SPARK-21563 Project: Spark

[jira] [Commented] (SPARK-14887) Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits

[ https://issues.apache.org/jira/browse/SPARK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099584#comment-16099584 ] Andrew Ash commented on SPARK-14887: [~fang fang chen] have you seen this in

[jira] [Commented] (SPARK-21289) Text and CSV formats do not support custom end-of-line delimiters

[ https://issues.apache.org/jira/browse/SPARK-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081136#comment-16081136 ] Andrew Ash commented on SPARK-21289: Looks like this will fix SPARK-17227

[jira] [Closed] (SPARK-15226) CSV file data-line with newline at first line load error

[ https://issues.apache.org/jira/browse/SPARK-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-15226. -- Resolution: Fixed Fix Version/s: 2.2.0 Fixed by Fixed by https://issues.apache.org/jira/browse

[jira] [Comment Edited] (SPARK-15226) CSV file data-line with newline at first line load error

[ https://issues.apache.org/jira/browse/SPARK-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081154#comment-16081154 ] Andrew Ash edited comment on SPARK-15226 at 7/10/17 9:0

[jira] [Created] (SPARK-21220) Use outputPartitioning's bucketing if possible on write

Andrew Ash created SPARK-21220: -- Summary: Use outputPartitioning's bucketing if possible on write Key: SPARK-21220 URL: https://issues.apache.org/jira/browse/SPARK-21220 Project: Spark

[jira] [Comment Edited] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058930#comment-16058930 ] Andrew Ash edited comment on SPARK-19700 at 6/22/17 7:4

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058930#comment-16058930 ] Andrew Ash commented on SPARK-19700: Public implementation that's been

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058917#comment-16058917 ] Andrew Ash commented on SPARK-19700: Found another potential implementa

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041803#comment-16041803 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: N

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035171#comment-16035171 ] Andrew Ash commented on SPARK-20952: For the localProperties on SparkContext it

[jira] [Created] (SPARK-20815) NullPointerException in RPackageUtils#checkManifestForR

Andrew Ash created SPARK-20815: -- Summary: NullPointerException in RPackageUtils#checkManifestForR Key: SPARK-20815 URL: https://issues.apache.org/jira/browse/SPARK-20815 Project: Spark Issue

[jira] [Commented] (SPARK-20683) Make table uncache chaining optional

[ https://issues.apache.org/jira/browse/SPARK-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018008#comment-16018008 ] Andrew Ash commented on SPARK-20683: Thanks for that diff [~shea.parkes] -- w

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012011#comment-16012011 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: E

[jira] [Created] (PARQUET-990) More detailed error messages in footer parsing

Andrew Ash created PARQUET-990: -- Summary: More detailed error messages in footer parsing Key: PARQUET-990 URL: https://issues.apache.org/jira/browse/PARQUET-990 Project: Parquet Issue Type

[jira] [Commented] (PARQUET-990) More detailed error messages in footer parsing

[ https://issues.apache.org/jira/browse/PARQUET-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011949#comment-16011949 ] Andrew Ash commented on PARQUET-990: https://github.com/apache/parquet-mr/pull

Re: [VOTE] Apache Spark 2.2.0 (RC1)

-1 due to regression from 2.1.1 In 2.2.0-rc1 we bumped the Parquet version from 1.8.1 to 1.8.2 in commit 26a4cba3ff . Parquet 1.8.2 includes a backport from 1.9.0: PARQUET-389 in commit 2282c22c

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979630#comment-15979630 ] Andrew Ash commented on SPARK-20433: It's unclear if Spark is affected, I

[jira] [Updated] (SPARK-20433) Security issue with jackson-databind

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Priority: Major (was: Blocker) > Security issue with jackson-datab

[jira] [Updated] (SPARK-20433) Security issue with jackson-databind

[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Priority: Critical (was: Major) > Security issue with jackson-datab

[jira] [Created] (SPARK-20433) Security issue with jackson-databind

Andrew Ash created SPARK-20433: -- Summary: Security issue with jackson-databind Key: SPARK-20433 URL: https://issues.apache.org/jira/browse/SPARK-20433 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973763#comment-15973763 ] Andrew Ash commented on SPARK-20364: Thanks for the investigation [~hyukjin.

[jira] [Comment Edited] (SPARK-1809) Mesos backend doesn't respect HADOOP_CONF_DIR

[ https://issues.apache.org/jira/browse/SPARK-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966782#comment-15966782 ] Andrew Ash edited comment on SPARK-1809 at 4/12/17 11:0

[jira] [Closed] (SPARK-1809) Mesos backend doesn't respect HADOOP_CONF_DIR

[ https://issues.apache.org/jira/browse/SPARK-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-1809. - Resolution: Unresolved Not using Mesos anymore, so closing > Mesos backend doesn't

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961170#comment-15961170 ] Andrew Ash commented on SPARK-20144: This is a regression from 1.6 to the 2.x

[jira] [Commented] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939072#comment-15939072 ] Andrew Ash commented on SPARK-19372: I've seen this as well on parq

[jira] [Updated] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-19528: --- Description: when dynamic allocation is enabled, the external shuffle service is used for maintain

[jira] [Updated] (SPARK-20001) Support PythonRunner executing inside a Conda env

[ https://issues.apache.org/jira/browse/SPARK-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20001: --- Description: Similar to SPARK-13587, I'm trying to allow the user to configure a Conda enviro

[jira] [Updated] (SPARK-20001) Support PythonRunner executing inside a Conda env

[ https://issues.apache.org/jira/browse/SPARK-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20001: --- Description: Similar to SPARK-13587, I'm trying to allow the user to configure a Conda enviro

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929591#comment-15929591 ] Andrew Ash commented on SPARK-18278: As an update on this ticket: For those

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893222#comment-15893222 ] Andrew Ash commented on SPARK-18113: We discovered another bug related to commit

[jira] [Commented] (SPARK-7354) Flaky test: o.a.s.deploy.SparkSubmitSuite --jars

[ https://issues.apache.org/jira/browse/SPARK-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881593#comment-15881593 ] Andrew Ash commented on SPARK-7354: --- We saw a flake for this test in the k8s re

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865778#comment-15865778 ] Andrew Ash commented on SPARK-18113: [~xukun] the scenario you describe shoul

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865536#comment-15865536 ] Andrew Ash commented on SPARK-18113: Thanks for the updates you both. I

[jira] [Commented] (SPARK-19493) Remove Java 7 support

[ https://issues.apache.org/jira/browse/SPARK-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861576#comment-15861576 ] Andrew Ash commented on SPARK-19493: +1 -- we're removing Java 7 compatibi

[jira] [Commented] (SPARK-11471) Improve the way that we plan shuffled join

[ https://issues.apache.org/jira/browse/SPARK-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15836547#comment-15836547 ] Andrew Ash commented on SPARK-11471: [~yhuai] I'm interested in hel

Re: Spark ANSI SQL Support

Rishabh, Have you come across any ANSI SQL queries that Spark SQL didn't support? I'd be interested to hear if you have. Andrew On Tue, Jan 17, 2017 at 8:14 PM, Deepak Sharma wrote: > From spark documentation page: > Spark SQL can now run all 99 TPC-DS queries. > > On Jan 18, 2017 9:39 AM, "Ri

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

[ https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824558#comment-15824558 ] Andrew Ash commented on YARN-2985: -- [~Steven Rand] I'm unable to attach patch

[jira] [Updated] (SPARK-19213) FileSourceScanExec uses SparkSession from HadoopFsRelation creation time instead of the active session at execution time

[ https://issues.apache.org/jira/browse/SPARK-19213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-19213: --- Summary: FileSourceScanExec uses SparkSession from HadoopFsRelation creation time instead of the

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812819#comment-15812819 ] Andrew Ash commented on SPARK-18113: I've done some more diagnosis on an

  1   2   3   4   5   6   7   8   9   10   >