[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-04-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14492303#comment-14492303 ] Steve Loughran commented on SPARK-1537: --- HADOOP-11826 patches the hadoop compatibili

[jira] [Commented] (SPARK-1911) Warn users if their assembly jars are not built with Java 6

2015-04-20 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502589#comment-14502589 ] Steve Loughran commented on SPARK-1911: --- This doesn't fix the problem, merely docume

[jira] [Created] (SPARK-7009) Build assembly JAR via ant to avoid zip64 problems

2015-04-20 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-7009: - Summary: Build assembly JAR via ant to avoid zip64 problems Key: SPARK-7009 URL: https://issues.apache.org/jira/browse/SPARK-7009 Project: Spark Issue Type

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

2015-06-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567345#comment-14567345 ] Steve Loughran commented on SPARK-4352: --- As usual, when YARN-1042 is done, life gets

[jira] [Commented] (SPARK-8064) Upgrade Hive to 1.2

2015-06-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575070#comment-14575070 ] Steve Loughran commented on SPARK-8064: --- I'm working on this > Upgrade Hive to 1.2

[jira] [Created] (SPARK-8275) HistoryServer caches incomplete App UIs

2015-06-09 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-8275: - Summary: HistoryServer caches incomplete App UIs Key: SPARK-8275 URL: https://issues.apache.org/jira/browse/SPARK-8275 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-8275) HistoryServer caches incomplete App UIs

2015-06-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579134#comment-14579134 ] Steve Loughran commented on SPARK-8275: --- I have a test for this as part of the SPARK

[jira] [Created] (SPARK-8276) NPE in YarnClientSchedulerBackend.stop

2015-06-09 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-8276: - Summary: NPE in YarnClientSchedulerBackend.stop Key: SPARK-8276 URL: https://issues.apache.org/jira/browse/SPARK-8276 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-06-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579152#comment-14579152 ] Steve Loughran commented on SPARK-1537: --- Full application log. Application hasn't ac

[jira] [Commented] (SPARK-8275) HistoryServer caches incomplete App UIs

2015-06-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579273#comment-14579273 ] Steve Loughran commented on SPARK-8275: --- The cache code is from google; history serv

[jira] [Comment Edited] (SPARK-8275) HistoryServer caches incomplete App UIs

2015-06-10 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579273#comment-14579273 ] Steve Loughran edited comment on SPARK-8275 at 6/10/15 11:43 AM: ---

[jira] [Issue Comment Deleted] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-06-10 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-1537: -- Comment: was deleted (was: Full application log. Application hasn't actually stopped, which is

[jira] [Resolved] (SPARK-3284) saveAsParquetFile not working on windows

2015-06-11 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-3284. --- Resolution: Duplicate closing as duplicate of SPARK-6961 > saveAsParquetFile not working on w

[jira] [Commented] (SPARK-6961) Cannot save data to parquet files when executing from Windows from a Maven Project

2015-06-11 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581886#comment-14581886 ] Steve Loughran commented on SPARK-6961: --- issue here is WINUTILS.EXE isn't on the pat

[jira] [Commented] (SPARK-8275) HistoryServer caches incomplete App UIs

2015-06-16 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587753#comment-14587753 ] Steve Loughran commented on SPARK-8275: --- your right, tagging as duplicate. > Histo

[jira] [Created] (SPARK-8394) HistoryServer doesn't read kerberos opts from config

2015-06-16 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-8394: - Summary: HistoryServer doesn't read kerberos opts from config Key: SPARK-8394 URL: https://issues.apache.org/jira/browse/SPARK-8394 Project: Spark Issue Ty

[jira] [Commented] (SPARK-8394) HistoryServer doesn't read kerberos opts from config

2015-06-16 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587882#comment-14587882 ] Steve Loughran commented on SPARK-8394: --- Fixed in commit [9042f](https://github.com

[jira] [Resolved] (SPARK-8394) HistoryServer doesn't read kerberos opts from config

2015-06-16 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-8394. --- Resolution: Fixed fixed by Marcelo > HistoryServer doesn't read kerberos opts from config > -

[jira] [Commented] (SPARK-7009) Build assembly JAR via ant to avoid zip64 problems

2015-06-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589659#comment-14589659 ] Steve Loughran commented on SPARK-7009: --- The issue isn't that python can't read larg

[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted

2015-06-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590215#comment-14590215 ] Steve Loughran commented on SPARK-7889: --- Is JIRA about (a) the status on the listin

[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted

2015-06-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595998#comment-14595998 ] Steve Loughran commented on SPARK-7889: --- Added a [new pull request|https://github.co

[jira] [Created] (SPARK-8789) improve SQLQuerySuite resilience by dropping tables in setup

2015-07-02 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-8789: - Summary: improve SQLQuerySuite resilience by dropping tables in setup Key: SPARK-8789 URL: https://issues.apache.org/jira/browse/SPARK-8789 Project: Spark

[jira] [Commented] (SPARK-15582) Support for Groovy closures

2016-06-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310096#comment-15310096 ] Steve Loughran commented on SPARK-15582: you are way ahead of me in your groovy s

[jira] [Commented] (SPARK-11695) Set s3a credentials by default similarly to s3 and s3n

2016-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321539#comment-15321539 ] Steve Loughran commented on SPARK-11695: There's some interesting ramifications t

[jira] [Created] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-15844: -- Summary: HistoryServer doesn't come up if spark.authenticate = true Key: SPARK-15844 URL: https://issues.apache.org/jira/browse/SPARK-15844 Project: Spark

[jira] [Commented] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322774#comment-15322774 ] Steve Loughran commented on SPARK-15844: Stack. {code} 16/05/31 22:46:25 INFO Sec

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

2016-06-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326356#comment-15326356 ] Steve Loughran commented on SPARK-2984: --- You cannot use speculation=true against S3

[jira] [Commented] (SPARK-15849) FileNotFoundException on _temporary while doing saveAsTable to S3

2016-06-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326391#comment-15326391 ] Steve Loughran commented on SPARK-15849: This sill be an S3 problem, quite a fund

[jira] [Commented] (SPARK-15923) Spark Application rest api returns "no such app: "

2016-06-14 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329276#comment-15329276 ] Steve Loughran commented on SPARK-15923: As thomas notes, it's probably not there

[jira] [Commented] (SPARK-15974) Create a socket on YARN AM start-up

2016-06-16 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333619#comment-15333619 ] Steve Loughran commented on SPARK-15974: # if someone hard codes a port in, that'

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2016-06-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15335878#comment-15335878 ] Steve Loughran commented on SPARK-15343: ooh, this is ia pain. FWIW, my current

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2019-02-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767077#comment-16767077 ] Steve Loughran commented on SPARK-23534: bq. am curious to know if hadoop3 offer

[jira] [Comment Edited] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2019-02-14 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201917#comment-16201917 ] Steve Loughran edited comment on SPARK-21797 at 2/14/19 3:54 PM: -

[jira] [Commented] (SPARK-25766) AMCredentialRenewer can leak FS clients

2019-02-14 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768747#comment-16768747 ] Steve Loughran commented on SPARK-25766: sounds good, though I should check. The

[jira] [Commented] (SPARK-27006) SPIP: .NET bindings for Apache Spark

2019-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786648#comment-16786648 ] Steve Loughran commented on SPARK-27006: I can see the appeal in having some ext

[jira] [Updated] (SPARK-27098) Flaky missing file parts when writing to Ceph without error

2019-03-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-27098: --- Description: https://stackoverflow.com/questions/54935822/spark-s3a-write-omits-upload-part-

[jira] [Commented] (SPARK-27098) Flaky missing file parts when writing to Ceph without error

2019-03-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787792#comment-16787792 ] Steve Loughran commented on SPARK-27098: It was my suggestion to file this. *

[jira] [Commented] (SPARK-27076) Getting the timeout error while writing parquet/csv files to s3

2019-03-15 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793868#comment-16793868 ] Steve Loughran commented on SPARK-27076: I've seen this before in the specific c

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-04-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449857#comment-16449857 ] Steve Loughran commented on SPARK-18673: It's a big hive patch, but most of it is

[jira] [Commented] (SPARK-24000) S3A: Create Table should fail on invalid AK/SK

2018-04-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450024#comment-16450024 ] Steve Loughran commented on SPARK-24000: We could consider whether or not to rais

[jira] [Updated] (SPARK-23654) Cut jets3t as a dependency of spark-core

2018-04-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23654: --- Summary: Cut jets3t as a dependency of spark-core (was: Cut jets3t as a dependency of spark-

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452196#comment-16452196 ] Steve Loughran commented on SPARK-18673: HIVE-16081 commit 93db527f47 contains th

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452215#comment-16452215 ] Steve Loughran commented on SPARK-18673: looking @ our local commit logs, the HDP

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2018-04-26 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453825#comment-16453825 ] Steve Loughran commented on SPARK-13446: [~Tavis]: can you paste in the stack you

[jira] [Commented] (SPARK-23151) Provide a distribution of Spark with Hadoop 3.0

2018-04-26 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453991#comment-16453991 ] Steve Loughran commented on SPARK-23151: well, there's "have everything work on H

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

2018-05-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465917#comment-16465917 ] Steve Loughran commented on SPARK-23977: It will need the hadoop-aws module and d

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-05-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465925#comment-16465925 ] Steve Loughran commented on SPARK-18673: Josh Rosen added some changes, particula

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-05-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465950#comment-16465950 ] Steve Loughran commented on SPARK-18673: Good Q, [~Bidek]. That SPARK-23807 POM f

[jira] [Created] (SPARK-24280) Speed up indexing of files in object stores by using listFiles(path, recursive=true)

2018-05-15 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-24280: -- Summary: Speed up indexing of files in object stores by using listFiles(path, recursive=true) Key: SPARK-24280 URL: https://issues.apache.org/jira/browse/SPARK-24280

[jira] [Commented] (SPARK-23681) Switch OrcFileFormat to newer hadoop.mapreduce output classes

2018-05-15 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475790#comment-16475790 ] Steve Loughran commented on SPARK-23681: sorry, been offline * yes, cut the versi

[jira] [Commented] (SPARK-19790) OutputCommitCoordinator should not allow another task to commit after an ExecutorFailure

2018-05-21 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482512#comment-16482512 ] Steve Loughran commented on SPARK-19790: Update on this, having spent lots of tim

[jira] [Commented] (SPARK-24271) sc.hadoopConfigurations can not be overwritten in the same spark context

2018-05-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16490474#comment-16490474 ] Steve Loughran commented on SPARK-24271: Disabling the s3 cache can be pretty ine

[jira] [Updated] (SPARK-24273) Failure while using .checkpoint method to private S3 store via S3A connector

2018-05-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24273: --- Summary: Failure while using .checkpoint method to private S3 store via S3A connector (was:

[jira] [Updated] (SPARK-24273) Failure while using .checkpoint method to private S3 store via S3A connector

2018-05-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24273: --- Description: We are getting following error: {code} com.amazonaws.services.s3.model.AmazonS3

[jira] [Commented] (SPARK-24273) Failure while using .checkpoint method to private S3 store via S3A connector

2018-05-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16492159#comment-16492159 ] Steve Loughran commented on SPARK-24273: Of course, there's no need to send range

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-06-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497928#comment-16497928 ] Steve Loughran commented on SPARK-18673: Jerry, I list up the other commits I'd

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

2018-06-04 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500560#comment-16500560 ] Steve Loughran commented on SPARK-20202: I think you could split things into two

[jira] [Created] (SPARK-24470) RestSubmissionClient to be robust against 404 & non json responses

2018-06-05 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-24470: -- Summary: RestSubmissionClient to be robust against 404 & non json responses Key: SPARK-24470 URL: https://issues.apache.org/jira/browse/SPARK-24470 Project: Spark

[jira] [Commented] (SPARK-24470) RestSubmissionClient to be robust against 404 & non json responses

2018-06-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502177#comment-16502177 ] Steve Loughran commented on SPARK-24470: stack from the issue {code} Running Spa

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24476: --- Component/s: (was: Spark Core) Structured Streaming > java.net.SocketTi

[jira] [Commented] (SPARK-24476) java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506002#comment-16506002 ] Steve Loughran commented on SPARK-24476: Switch from s3n to the s3a connector, s

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24476: --- Summary: java.net.SocketTimeoutException: Read timed out under jets3t while running the Spar

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24476: --- Priority: Minor (was: Major) > java.net.SocketTimeoutException: Read timed out under jets3t

[jira] [Updated] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24492: --- Summary: Endless attempted task when TaskCommitDenied exception writing to S3A (was: Endles

[jira] [Commented] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506009#comment-16506009 ] Steve Loughran commented on SPARK-24492: the retry problem looks like something

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506021#comment-16506021 ] Steve Loughran commented on SPARK-23534: [~jerryshao] is that the HDFS token ide

[jira] [Commented] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

2018-06-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507113#comment-16507113 ] Steve Loughran commented on SPARK-24492: well, you've got a consistency problem

[jira] [Commented] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

2018-06-15 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16513933#comment-16513933 ] Steve Loughran commented on SPARK-24476: * Use S3A, as S3n is unsupported and de

[jira] [Commented] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

2018-07-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534725#comment-16534725 ] Steve Loughran commented on SPARK-24492: I think you'll have to set the logs to

[jira] [Commented] (SPARK-24746) AWS S3 301 Moved Permanently error message even after setting fs.s3a.endpoint for bucket in Mumbai region.

2018-07-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534728#comment-16534728 ] Steve Loughran commented on SPARK-24746: Mumbai is v4 auth, which isn't directly

[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

2018-07-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541915#comment-16541915 ] Steve Loughran commented on SPARK-21962: Yes, assume that: * Htrace goes off t

[jira] [Created] (SPARK-25766) AMCredentialRenewer can leak FS clients

2018-10-18 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-25766: -- Summary: AMCredentialRenewer can leak FS clients Key: SPARK-25766 URL: https://issues.apache.org/jira/browse/SPARK-25766 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2018-10-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656476#comment-16656476 ] Steve Loughran commented on SPARK-19143: I'm looking at this as I add DTs into S

[jira] [Comment Edited] (SPARK-21725) spark thriftserver insert overwrite table partition select

2018-10-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656631#comment-16656631 ] Steve Loughran edited comment on SPARK-21725 at 10/19/18 10:48 AM: ---

[jira] [Commented] (SPARK-21725) spark thriftserver insert overwrite table partition select

2018-10-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656631#comment-16656631 ] Steve Loughran commented on SPARK-21725: bq. can we fix it on the Hadoop side?

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2018-10-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657214#comment-16657214 ] Steve Loughran commented on SPARK-19143: thanks. when I get that S3a patch in s

[jira] [Commented] (SPARK-25855) Don't use Erasure Coding for event log files

2018-10-29 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667270#comment-16667270 ] Steve Loughran commented on SPARK-25855: thx for the mention. yes, looking @ str

[jira] [Commented] (SPARK-25966) "EOF Reached the end of stream with bytes left to read" while reading/writing to Parquets

2018-11-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679611#comment-16679611 ] Steve Loughran commented on SPARK-25966: bq. It looks to me like a problem in c

[jira] [Commented] (SPARK-25966) "EOF Reached the end of stream with bytes left to read" while reading/writing to Parquets

2018-11-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681297#comment-16681297 ] Steve Loughran commented on SPARK-25966: bq. Hadoop 3.1.x is not yet officially

[jira] [Commented] (SPARK-26188) Spark 2.4.0 Partitioning behavior breaks backwards compatibility

2018-11-30 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704804#comment-16704804 ] Steve Loughran commented on SPARK-26188: bq. > My team uses spark to partition a

[jira] [Commented] (SPARK-26254) Move delegation token providers into a separate project

2018-12-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707679#comment-16707679 ] Steve Loughran commented on SPARK-26254: +HBase I don't have any opinions on th

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

2017-11-16 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255329#comment-16255329 ] Steve Loughran commented on SPARK-16996: [~maver1ck] Spark hive is custom as it

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-11-16 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255861#comment-16255861 ] Steve Loughran commented on SPARK-22240: I think there are two separate issues an

[jira] [Commented] (SPARK-14959) ​Problem Reading partitioned ORC or Parquet files

2017-11-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256789#comment-16256789 ] Steve Loughran commented on SPARK-14959: Came across a reference to this while sc

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-21 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261292#comment-16261292 ] Steve Loughran commented on SPARK-22526: S3a uses the AWS S3 client, which uses h

[jira] [Commented] (SPARK-22374) STS ran into OOM in a secure cluster

2017-11-21 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261444#comment-16261444 ] Steve Loughran commented on SPARK-22374: We need to do something about this, it i

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263021#comment-16263021 ] Steve Loughran commented on SPARK-22526: If the input stream doesn't get closed,

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264095#comment-16264095 ] Steve Loughran commented on SPARK-22526: # Fix the code you invoke #. wrap the co

[jira] [Comment Edited] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264095#comment-16264095 ] Steve Loughran edited comment on SPARK-22526 at 11/23/17 3:47 PM: -

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264496#comment-16264496 ] Steve Loughran commented on SPARK-22526: I'm not giving a permanent fix. It's a b

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265167#comment-16265167 ] Steve Loughran commented on SPARK-22526: it says it in the javadocs for [Portable

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-11-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266674#comment-16266674 ] Steve Loughran commented on SPARK-22587: Jerry had already pulled me in for this;

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-11-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266697#comment-16266697 ] Steve Loughran commented on SPARK-22587: See also [FileSystem.CACHE.Key.isEqual(

[jira] [Commented] (SPARK-22526) Document closing of PortableDataInputStream in binaryFiles

2017-11-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16267015#comment-16267015 ] Steve Loughran commented on SPARK-22526: HADOOP-15071 updates the s3a troubleshoo

[jira] [Commented] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-30 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272974#comment-16272974 ] Steve Loughran commented on SPARK-22657: Hadoop FileSystem service introspection

[jira] [Commented] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-30 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273191#comment-16273191 ] Steve Loughran commented on SPARK-22657: if you look at HADOOP-14138 you can see

[jira] [Commented] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-12-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274310#comment-16274310 ] Steve Loughran commented on SPARK-22657: No, more that we need to change how that

[jira] [Commented] (SPARK-21074) Parquet files are read fully even though only count() is requested

2017-12-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281901#comment-16281901 ] Steve Loughran commented on SPARK-21074: Is there any update on this? # I'd lik

[jira] [Commented] (SPARK-18294) Implement commit protocol to support `mapred` package's committer

2017-12-15 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292595#comment-16292595 ] Steve Loughran commented on SPARK-18294: Following up on this, one question: Why

[jira] [Commented] (SPARK-7755) MetadataCache.refresh does not take into account _SUCCESS

2018-01-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313803#comment-16313803 ] Steve Loughran commented on SPARK-7755: --- I concur with [~hyukjin.kwon] here: if inco

<    1   2   3   4   5   6   7   8   9   >