[jira] [Commented] (SPARK-21077) Cannot access public files over S3 protocol

[ https://issues.apache.org/jira/browse/SPARK-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051680#comment-16051680 ] Steve Loughran commented on SPARK-21077: like people say, this is inevitably a co

[jira] [Commented] (SPARK-21074) Parquet files are read fully even though only count() is requested

[ https://issues.apache.org/jira/browse/SPARK-21074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056309#comment-16056309 ] Steve Loughran commented on SPARK-21074: Given this is an s3 URL, it may be ampli

[jira] [Commented] (SPARK-19111) S3 Mesos history upload fails silently if too large

[ https://issues.apache.org/jira/browse/SPARK-19111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056516#comment-16056516 ] Steve Loughran commented on SPARK-19111: Followup: [~drcrallen]; Hadoop 2.8 is ou

[jira] [Commented] (SPARK-11373) Add metrics to the History Server and providers

[ https://issues.apache.org/jira/browse/SPARK-11373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059141#comment-16059141 ] Steve Loughran commented on SPARK-11373: metrics might help with understanding th

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065423#comment-16065423 ] Steve Loughran commented on SPARK-21137: Looking at this. something is trying to

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065428#comment-16065428 ] Steve Loughran commented on SPARK-21137: ps, for now, do it in parallel: {{mapre

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065445#comment-16065445 ] Steve Loughran commented on SPARK-21137: Filed HADOOP-14600. Looks like a v. old

[jira] [Commented] (SPARK-12868) ADD JAR via sparkSQL JDBC will fail when using a HDFS URL

[ https://issues.apache.org/jira/browse/SPARK-12868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065448#comment-16065448 ] Steve Loughran commented on SPARK-12868: I think this is the case of HADOOP-14598

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065515#comment-16065515 ] Steve Loughran commented on SPARK-21137: bq. so it is something that could be opt

[jira] [Updated] (SPARK-21137) Spark reads many small files slowly off local filesystem

[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-21137: --- Summary: Spark reads many small files slowly off local filesystem (was: Spark reads many sma

[jira] [Commented] (SPARK-12868) ADD JAR via sparkSQL JDBC will fail when using a HDFS URL

[ https://issues.apache.org/jira/browse/SPARK-12868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068301#comment-16068301 ] Steve Loughran commented on SPARK-12868: It's actually not the cause of that, mer

[jira] [Commented] (SPARK-20107) Add spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version option to configuration.md

[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073951#comment-16073951 ] Steve Loughran commented on SPARK-20107: If you are curious, I've just written ou

[jira] [Commented] (SPARK-20703) Add an operator for writing data out

[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084355#comment-16084355 ] Steve Loughran commented on SPARK-20703: this has just added a whole new stack tr

[jira] [Commented] (SPARK-20703) Add an operator for writing data out

[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085488#comment-16085488 ] Steve Loughran commented on SPARK-20703: yeah, I'm not worrying too much about th

[jira] [Commented] (SPARK-20703) Add an operator for writing data out

[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085559#comment-16085559 ] Steve Loughran commented on SPARK-20703: Regarding a patch for this, what do peop

[jira] [Commented] (SPARK-21374) Reading globbed paths from S3 into DF doesn't work if filesystem caching is disabled

[ https://issues.apache.org/jira/browse/SPARK-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085645#comment-16085645 ] Steve Loughran commented on SPARK-21374: This is possibly a sign that your new co

[jira] [Commented] (SPARK-19790) OutputCommitCoordinator should not allow another task to commit after an ExecutorFailure

[ https://issues.apache.org/jira/browse/SPARK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088630#comment-16088630 ] Steve Loughran commented on SPARK-19790: I've now summarised the FileOutputCommit

[jira] [Commented] (SPARK-20703) Add an operator for writing data out

[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088632#comment-16088632 ] Steve Loughran commented on SPARK-20703: ..got a patch for this, but want to see

[jira] [Commented] (SPARK-21514) Hive has updated with new support for S3 and InsertIntoHiveTable.scala should update also

[ https://issues.apache.org/jira/browse/SPARK-21514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108882#comment-16108882 ] Steve Loughran commented on SPARK-21514: Can you link this JIRA to the specific H

[jira] [Commented] (SPARK-21374) Reading globbed paths from S3 into DF doesn't work if filesystem caching is disabled

[ https://issues.apache.org/jira/browse/SPARK-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108897#comment-16108897 ] Steve Loughran commented on SPARK-21374: I understand...the patch shows the issue

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112511#comment-16112511 ] Steve Loughran commented on SPARK-21618: It may depend on HADOOP-14383; I wouldn'

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112520#comment-16112520 ] Steve Loughran commented on SPARK-21618: BTW, we haven't backported HADOOP-14383

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112529#comment-16112529 ] Steve Loughran commented on SPARK-21618: If you're relying on hadoop-common to pr

[jira] [Comment Edited] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112529#comment-16112529 ] Steve Loughran edited comment on SPARK-21618 at 8/3/17 10:09 AM: --

[jira] [Commented] (SPARK-15923) Spark Application rest api returns "no such app: "

[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370547#comment-15370547 ] Steve Loughran commented on SPARK-15923: I do think the docs could be clarified a

[jira] [Commented] (SPARK-13514) Spark Shuffle Service 1.6.0 issue in Yarn

[ https://issues.apache.org/jira/browse/SPARK-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372728#comment-15372728 ] Steve Loughran commented on SPARK-13514: does the same file:// config setting exi

[jira] [Commented] (SPARK-7481) Add spark-cloud module to pull in aws+azure object store FS accessors; test integration

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389707#comment-15389707 ] Steve Loughran commented on SPARK-7481: --- Sad but true. * The PR I've put up adds th

[jira] [Commented] (SPARK-7481) Add spark-cloud module to pull in aws+azure object store FS accessors; test integration

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389730#comment-15389730 ] Steve Loughran commented on SPARK-7481: --- ps, latest s3a state # [Object stores in

[jira] [Created] (SPARK-16736) remove redundant FileSystem.exists() calls from Spark codebase

Steve Loughran created SPARK-16736: -- Summary: remove redundant FileSystem.exists() calls from Spark codebase Key: SPARK-16736 URL: https://issues.apache.org/jira/browse/SPARK-16736 Project: Spark

[jira] [Updated] (SPARK-16737) ListingFileCatalog comments about RPC calls in object store isn't correct

[ https://issues.apache.org/jira/browse/SPARK-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-16737: --- Description: The comment text which came in with SPARK-16121 says {code} - Although S3/S3A/

[jira] [Created] (SPARK-16737) ListingFileCatalog comments about RPC calls in object store isn't correct

Steve Loughran created SPARK-16737: -- Summary: ListingFileCatalog comments about RPC calls in object store isn't correct Key: SPARK-16737 URL: https://issues.apache.org/jira/browse/SPARK-16737 Project

[jira] [Updated] (SPARK-16736) remove redundant FileSystem status checks calls from Spark codebase

[ https://issues.apache.org/jira/browse/SPARK-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-16736: --- Summary: remove redundant FileSystem status checks calls from Spark codebase (was: remove re

[jira] [Commented] (SPARK-16736) remove redundant FileSystem status checks calls from Spark codebase

[ https://issues.apache.org/jira/browse/SPARK-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393770#comment-15393770 ] Steve Loughran commented on SPARK-16736: See also HIVE-14323 > remove redundant

[jira] [Commented] (SPARK-27006) SPIP: .NET bindings for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-27006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786648#comment-16786648 ] Steve Loughran commented on SPARK-27006: I can see the appeal in having some ext

[jira] [Updated] (SPARK-27098) Flaky missing file parts when writing to Ceph without error

[ https://issues.apache.org/jira/browse/SPARK-27098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-27098: --- Description: https://stackoverflow.com/questions/54935822/spark-s3a-write-omits-upload-part-

[jira] [Commented] (SPARK-27098) Flaky missing file parts when writing to Ceph without error

[ https://issues.apache.org/jira/browse/SPARK-27098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787792#comment-16787792 ] Steve Loughran commented on SPARK-27098: It was my suggestion to file this. *

[jira] [Commented] (SPARK-27076) Getting the timeout error while writing parquet/csv files to s3

[ https://issues.apache.org/jira/browse/SPARK-27076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793868#comment-16793868 ] Steve Loughran commented on SPARK-27076: I've seen this before in the specific c

[jira] [Commented] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8.2

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829155#comment-16829155 ] Steve Loughran commented on SPARK-24771: Update Hadoop is going to update its av

[jira] [Commented] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15591647#comment-15591647 ] Steve Loughran commented on SPARK-10673: This may be related to SPARK-17179; Hive

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602160#comment-15602160 ] Steve Loughran commented on SPARK-2984: --- Alexy, can you describe your layout a bit m

[jira] [Commented] (SPARK-14222) Cross-publish jackson-module-scala for Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15635835#comment-15635835 ] Steve Loughran commented on SPARK-14222: In Hadoop we're looking at -> 2.7.x to (

[jira] [Commented] (SPARK-7344) Spark hangs reading and writing to the same S3 bucket

[ https://issues.apache.org/jira/browse/SPARK-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643819#comment-15643819 ] Steve Loughran commented on SPARK-7344: --- I've been doing lots of work with S3a and n

[jira] [Commented] (SPARK-13044) saveAsTextFile() doesn't support s3 Signature Version 4

[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643827#comment-15643827 ] Steve Loughran commented on SPARK-13044: This is HADOOP-13325; jets3t doesn't sup

[jira] [Updated] (SPARK-13044) saveAsTextFile(s3n://) doesn't support s3 Signature Version 4

[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-13044: --- Summary: saveAsTextFile(s3n://) doesn't support s3 Signature Version 4 (was: saveAsTextFile(

[jira] [Resolved] (SPARK-13044) saveAsTextFile(s3n://) doesn't support s3 Signature Version 4

[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-13044. Resolution: Won't Fix > saveAsTextFile(s3n://) doesn't support s3 Signature Version 4 > ---

[jira] [Resolved] (SPARK-12378) CREATE EXTERNAL TABLE AS SELECT EXPORT AWS S3 ERROR

[ https://issues.apache.org/jira/browse/SPARK-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-12378. Resolution: Cannot Reproduce > CREATE EXTERNAL TABLE AS SELECT EXPORT AWS S3 ERROR > --

[jira] [Commented] (SPARK-12378) CREATE EXTERNAL TABLE AS SELECT EXPORT AWS S3 ERROR

[ https://issues.apache.org/jira/browse/SPARK-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643851#comment-15643851 ] Steve Loughran commented on SPARK-12378: This is amazon EMR; they've got their ow

[jira] [Commented] (SPARK-18017) Changing Hadoop parameter through sparkSession.sparkContext.hadoopConfiguration doesn't work

[ https://issues.apache.org/jira/browse/SPARK-18017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643864#comment-15643864 ] Steve Loughran commented on SPARK-18017: you can check what's been picked up by g

[jira] [Commented] (SPARK-10063) Remove DirectParquetOutputCommitter

[ https://issues.apache.org/jira/browse/SPARK-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644092#comment-15644092 ] Steve Loughran commented on SPARK-10063: HADOOP-13786 covers adding a committer f

[jira] [Commented] (SPARK-18402) spark: SAXParseException while writing from json to parquet on s3

[ https://issues.apache.org/jira/browse/SPARK-18402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657769#comment-15657769 ] Steve Loughran commented on SPARK-18402: I've seen this before, somewhere. it's u

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679196#comment-15679196 ] Steve Loughran commented on SPARK-2984: --- That sounds like a separate issue...could y

[jira] [Commented] (SPARK-14222) Cross-publish jackson-module-scala for Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15683998#comment-15683998 ] Steve Loughran commented on SPARK-14222: Hadoop 2.9 just went to Java 2.7.8; late

[jira] [Commented] (SPARK-6527) sc.binaryFiles can not access files on s3

[ https://issues.apache.org/jira/browse/SPARK-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259963#comment-15259963 ] Steve Loughran commented on SPARK-6527: --- I've not seen a JIRA surface; # if anyone

[jira] [Commented] (SPARK-6527) sc.binaryFiles can not access files on s3

[ https://issues.apache.org/jira/browse/SPARK-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259966#comment-15259966 ] Steve Loughran commented on SPARK-6527: --- Actually, looking at {{SparkContext.binaryF

[jira] [Created] (SPARK-15090) Spark Hive thriftserver can get 413 errors in Kerberos+AD deployments

Steve Loughran created SPARK-15090: -- Summary: Spark Hive thriftserver can get 413 errors in Kerberos+AD deployments Key: SPARK-15090 URL: https://issues.apache.org/jira/browse/SPARK-15090 Project: Sp

[jira] [Updated] (SPARK-7481) Add spark-cloud module to pull in aws+azure object store FS accessors; test integration

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-7481: -- Summary: Add spark-cloud module to pull in aws+azure object store FS accessors; test integration

[jira] [Commented] (SPARK-7481) Add spark-cloud module to pull in aws+azure object store FS accessors; test integration

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274218#comment-15274218 ] Steve Loughran commented on SPARK-7481: --- For people watching this, know that it work

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15286954#comment-15286954 ] Steve Loughran commented on SPARK-15343: thanks for this, I'll look at it. FWIW i

[jira] [Commented] (SPARK-13599) Groovy-all ends up in spark-assembly if hive profile set

[ https://issues.apache.org/jira/browse/SPARK-13599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299793#comment-15299793 ] Steve Loughran commented on SPARK-13599: Sorry to hear about this: I know precise

[jira] [Commented] (SPARK-13599) Groovy-all ends up in spark-assembly if hive profile set

[ https://issues.apache.org/jira/browse/SPARK-13599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299800#comment-15299800 ] Steve Loughran commented on SPARK-13599: correction: if you have groovy 2.4.6 you

[jira] [Commented] (SPARK-33605) Add GCS FS/connector config (dependencies?) akin to S3

[ https://issues.apache.org/jira/browse/SPARK-33605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267398#comment-17267398 ] Steve Loughran commented on SPARK-33605: hadoop-aws and aws-sdk JARs come if you

[jira] [Commented] (SPARK-32582) Spark SQL Infer Schema Performance

[ https://issues.apache.org/jira/browse/SPARK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267408#comment-17267408 ] Steve Loughran commented on SPARK-32582: Returning to this. The incremental li

[jira] [Commented] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270263#comment-17270263 ] Steve Loughran commented on SPARK-34194: * if you can also use any of the increm

[jira] [Commented] (SPARK-34298) SaveMode.Overwrite not usable when using s3a root paths

[ https://issues.apache.org/jira/browse/SPARK-34298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278846#comment-17278846 ] Steve Loughran commented on SPARK-34298: root dirs are special in that they alwa

[jira] [Commented] (SPARK-34298) SaveMode.Overwrite not usable when using s3a root paths

[ https://issues.apache.org/jira/browse/SPARK-34298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279798#comment-17279798 ] Steve Loughran commented on SPARK-34298: well, I'm sure a PR with tests will get

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309397#comment-17309397 ] Steve Loughran commented on SPARK-23977: use the partitioned committer and confi

[jira] [Commented] (SPARK-36766) Spark SQL DDL does not recognize fs.s3.impl implied filesystem in LOCATION tag

[ https://issues.apache.org/jira/browse/SPARK-36766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426152#comment-17426152 ] Steve Loughran commented on SPARK-36766: I can see why you'd want to do this (co

[jira] [Commented] (SPARK-36529) Decouple CPU with IO work in vectorized Parquet reader

[ https://issues.apache.org/jira/browse/SPARK-36529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426158#comment-17426158 ] Steve Loughran commented on SPARK-36529: If you look at HADOOP-11867 / https://g

[jira] [Commented] (SPARK-35428) Spark history Server to S3 doesn't show incomplete applications

[ https://issues.apache.org/jira/browse/SPARK-35428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426159#comment-17426159 ] Steve Loughran commented on SPARK-35428: # please stop using s3n; that connector

[jira] [Commented] (SPARK-36761) spark-examples_2.12-3.0.2.jar DFSReadWriteTest S3A Implementation

[ https://issues.apache.org/jira/browse/SPARK-36761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426163#comment-17426163 ] Steve Loughran commented on SPARK-36761: something in the code has got the defau

[jira] [Commented] (SPARK-36024) Switch the datasource example due to the depreciation of the dataset

[ https://issues.apache.org/jira/browse/SPARK-36024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426166#comment-17426166 ] Steve Loughran commented on SPARK-36024: Amazon are being very nice here and kee

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17437259#comment-17437259 ] Steve Loughran commented on SPARK-23977: [~gumartinm] can I draw your attention

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510830#comment-17510830 ] Steve Loughran commented on SPARK-38330: the hadoop fix is in, but it will take

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512458#comment-17512458 ] Steve Loughran commented on SPARK-38652: have you tried running the same suite a

[jira] [Commented] (SPARK-38445) Are hadoop committers used in Structured Streaming?

[ https://issues.apache.org/jira/browse/SPARK-38445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517556#comment-17517556 ] Steve Loughran commented on SPARK-38445: not suppoorted unless you provide the P

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17523891#comment-17523891 ] Steve Loughran commented on SPARK-38330: FWIW I'm not 100% sure this is fixed, a

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17525719#comment-17525719 ] Steve Loughran commented on SPARK-38330: aws sdk does its own thing sometimes, f

[jira] [Commented] (SPARK-38954) Implement sharing of cloud credentials among driver and executors

[ https://issues.apache.org/jira/browse/SPARK-38954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541002#comment-17541002 ] Steve Loughran commented on SPARK-38954: what is the strategy for having the wor

[jira] [Commented] (SPARK-29250) Upgrade to Hadoop 3.3.1

[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17553744#comment-17553744 ] Steve Loughran commented on SPARK-29250: use whatever version the spark release

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314296#comment-17314296 ] Steve Loughran commented on SPARK-23977: the spark settings don't make it down f

[jira] [Commented] (SPARK-34925) Spark shell failed to access external file to read content

[ https://issues.apache.org/jira/browse/SPARK-34925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320317#comment-17320317 ] Steve Loughran commented on SPARK-34925: you've got an s3a config option set to

[jira] [Commented] (SPARK-26284) Spark History server object vs file storage behavior difference

[ https://issues.apache.org/jira/browse/SPARK-26284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17347496#comment-17347496 ] Steve Loughran commented on SPARK-26284: # s3:// URLs mean you are using EMR? If

[jira] [Commented] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350371#comment-17350371 ] Steve Loughran commented on SPARK-35406: This is actually triggered by garbage c

[jira] [Commented] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350372#comment-17350372 ] Steve Loughran commented on SPARK-35299: +use a version of spark with the hadoop

[jira] [Commented] (SPARK-35279) _SUCCESS file not written when using partitionOverwriteMode=dynamic

[ https://issues.apache.org/jira/browse/SPARK-35279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350373#comment-17350373 ] Steve Loughran commented on SPARK-35279: Any reason for not using the S3A commit

[jira] [Commented] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356607#comment-17356607 ] Steve Loughran commented on SPARK-35406: This is fixed by HADOOP-17338 and by th

[jira] [Updated] (SPARK-35299) Dataframe overwrite on S3A does not delete old files with S3 object-put to table path/

[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-35299: --- Summary: Dataframe overwrite on S3A does not delete old files with S3 object-put to table pa

[jira] [Commented] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356615#comment-17356615 ] Steve Loughran commented on SPARK-35299: Actually if you are doing a PUT of test

[jira] [Commented] (SPARK-34298) SaveMode.Overwrite not usable when using s3a root paths

[ https://issues.apache.org/jira/browse/SPARK-34298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356625#comment-17356625 ] Steve Loughran commented on SPARK-34298: Actually, you've just found a bug in Pa

[jira] [Resolved] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-35406. Resolution: Done > TaskCompletionListenerException: Premature end of Content-Length delimi

[jira] [Commented] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357213#comment-17357213 ] Steve Loughran commented on SPARK-35406: bq. This backport of Hadoop-12444(lazy

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466876#comment-17466876 ] Steve Loughran commented on SPARK-6305: --- If anyone wants a version of a log4j 1.17

[jira] [Commented] (SPARK-37630) Security issue from Log4j 1.X exploit

[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466918#comment-17466918 ] Steve Loughran commented on SPARK-37630: nobody does. you can find a patched ja

[jira] [Comment Edited] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466876#comment-17466876 ] Steve Loughran edited comment on SPARK-6305 at 12/30/21, 6:44 PM: -

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469178#comment-17469178 ] Steve Loughran commented on SPARK-37814: be good to link to all issues related t

[jira] [Commented] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470842#comment-17470842 ] Steve Loughran commented on SPARK-37771: probably related to HADOOP-17372, which

[jira] [Commented] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485973#comment-17485973 ] Steve Loughran commented on SPARK-37771: [~ivan.sadikov] -any update here? > Ra

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488804#comment-17488804 ] Steve Loughran commented on SPARK-37814: everyone is aware of the log4j issues,

[jira] [Comment Edited] (SPARK-37814) Migrating from log4j 1 to log4j 2

[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488804#comment-17488804 ] Steve Loughran edited comment on SPARK-37814 at 2/8/22, 12:04 PM:

[jira] [Commented] (SPARK-38115) No spark conf to control the path of _temporary when writing to target filesystem

[ https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492810#comment-17492810 ] Steve Loughran commented on SPARK-38115: * stop using the classic FileOutputComm

  1   2   3   4   5   6   7   8   9   >