[jira] [Commented] (SPARK-38115) No spark conf to control the path of _temporary when writing to target filesystem

[ https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17496022#comment-17496022 ] Steve Loughran commented on SPARK-38115: bq. Is there any config as such to stop

[jira] [Created] (SPARK-38394) build of spark sql against hadoop-3.4.0-snapshot failing with bouncycastle classpath error

Steve Loughran created SPARK-38394: -- Summary: build of spark sql against hadoop-3.4.0-snapshot failing with bouncycastle classpath error Key: SPARK-38394 URL: https://issues.apache.org/jira/browse/SPARK-38394

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504273#comment-17504273 ] Steve Loughran commented on SPARK-31911: I'm going to close as fixed now; the sp

[jira] [Resolved] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-31911. Fix Version/s: 3.0.1 2.4.7 Resolution: Fixed > Using S3A staging

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504286#comment-17504286 ] Steve Loughran commented on SPARK-38330: this is a hadoop issue -create a Jira t

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507557#comment-17507557 ] Steve Loughran commented on SPARK-38330: sorry about that. try enabling path sty

[jira] [Updated] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-22163: --- Priority: Major (was: Critical) > Design Issue of Spark Streaming that Causes Random Run-tim

[jira] [Commented] (SPARK-21999) ConcurrentModificationException - Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192732#comment-16192732 ] Steve Loughran commented on SPARK-21999: Apache projects are all open source, wit

[jira] [Comment Edited] (SPARK-21999) ConcurrentModificationException - Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192732#comment-16192732 ] Steve Loughran edited comment on SPARK-21999 at 10/5/17 1:39 PM: --

[jira] [Commented] (SPARK-21999) ConcurrentModificationException - Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194332#comment-16194332 ] Steve Loughran commented on SPARK-21999: Telling a project "their design is wrong

[jira] [Resolved] (SPARK-21999) ConcurrentModificationException - Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-21999. Resolution: Won't Fix > ConcurrentModificationException - Spark Streaming > ---

[jira] [Created] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters if parquet.enable.summary-metadata is false

Steve Loughran created SPARK-22217: -- Summary: ParquetFileFormat to support arbitrary OutputCommitters if parquet.enable.summary-metadata is false Key: SPARK-22217 URL: https://issues.apache.org/jira/browse/SPARK-

[jira] [Updated] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-22217: --- Summary: ParquetFileFormat to support arbitrary OutputCommitters (was: ParquetFileFormat to

[jira] [Updated] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-22217: --- Priority: Minor (was: Major) > ParquetFileFormat to support arbitrary OutputCommitters > ---

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200367#comment-16200367 ] Steve Loughran commented on SPARK-22240: Amazon EMR is amazon's own fork of Spark

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200485#comment-16200485 ] Steve Loughran commented on SPARK-22240: What's the link to the multiline JIRA? A

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201863#comment-16201863 ] Steve Loughran commented on SPARK-22240: thanks. Now for a question which is prob

[jira] [Commented] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201917#comment-16201917 ] Steve Loughran commented on SPARK-21797: Update, in HADOOP-14874 I've noted we co

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203495#comment-16203495 ] Steve Loughran commented on SPARK-22240: We've got a test in HADOOP-14943 which l

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203709#comment-16203709 ] Steve Loughran commented on SPARK-22240: [~hyukjin.kwon]: we now see that on s3a,

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203966#comment-16203966 ] Steve Loughran commented on SPARK-22240: Point me at a simple test suite for the

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207573#comment-16207573 ] Steve Loughran commented on SPARK-2984: --- bq. multiple batches writing to same locati

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217325#comment-16217325 ] Steve Loughran commented on SPARK-22240: I'm doing some testing with master & rea

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217398#comment-16217398 ] Steve Loughran commented on SPARK-22240: no, spark 2.2 doesn't fix this. I have

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223400#comment-16223400 ] Steve Loughran commented on SPARK-22240: so this partition calculation problem is

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233924#comment-16233924 ] Steve Loughran commented on SPARK-2984: --- Darron: different stack trace, different pa

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233937#comment-16233937 ] Steve Loughran commented on SPARK-2984: --- [~soumdmw] you asked bq. is there a simple

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247886#comment-16247886 ] Steve Loughran commented on SPARK-17593: Hey nick, yes, need to move to FileSys

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255329#comment-16255329 ] Steve Loughran commented on SPARK-16996: [~maver1ck] Spark hive is custom as it

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255861#comment-16255861 ] Steve Loughran commented on SPARK-22240: I think there are two separate issues an

[jira] [Commented] (SPARK-14959) ​Problem Reading partitioned ORC or Parquet files

[ https://issues.apache.org/jira/browse/SPARK-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256789#comment-16256789 ] Steve Loughran commented on SPARK-14959: Came across a reference to this while sc

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261292#comment-16261292 ] Steve Loughran commented on SPARK-22526: S3a uses the AWS S3 client, which uses h

[jira] [Commented] (SPARK-22374) STS ran into OOM in a secure cluster

[ https://issues.apache.org/jira/browse/SPARK-22374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261444#comment-16261444 ] Steve Loughran commented on SPARK-22374: We need to do something about this, it i

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263021#comment-16263021 ] Steve Loughran commented on SPARK-22526: If the input stream doesn't get closed,

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264095#comment-16264095 ] Steve Loughran commented on SPARK-22526: # Fix the code you invoke #. wrap the co

[jira] [Comment Edited] (SPARK-22526) Spark hangs while reading binary files from S3

[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264095#comment-16264095 ] Steve Loughran edited comment on SPARK-22526 at 11/23/17 3:47 PM: -

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264496#comment-16264496 ] Steve Loughran commented on SPARK-22526: I'm not giving a permanent fix. It's a b

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265167#comment-16265167 ] Steve Loughran commented on SPARK-22526: it says it in the javadocs for [Portable

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266674#comment-16266674 ] Steve Loughran commented on SPARK-22587: Jerry had already pulled me in for this;

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266697#comment-16266697 ] Steve Loughran commented on SPARK-22587: See also [FileSystem.CACHE.Key.isEqual(

[jira] [Commented] (SPARK-22526) Document closing of PortableDataInputStream in binaryFiles

[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16267015#comment-16267015 ] Steve Loughran commented on SPARK-22526: HADOOP-15071 updates the s3a troubleshoo

[jira] [Commented] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272974#comment-16272974 ] Steve Loughran commented on SPARK-22657: Hadoop FileSystem service introspection

[jira] [Commented] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273191#comment-16273191 ] Steve Loughran commented on SPARK-22657: if you look at HADOOP-14138 you can see

[jira] [Commented] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274310#comment-16274310 ] Steve Loughran commented on SPARK-22657: No, more that we need to change how that

[jira] [Commented] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534725#comment-16534725 ] Steve Loughran commented on SPARK-24492: I think you'll have to set the logs to

[jira] [Commented] (SPARK-24746) AWS S3 301 Moved Permanently error message even after setting fs.s3a.endpoint for bucket in Mumbai region.

[ https://issues.apache.org/jira/browse/SPARK-24746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534728#comment-16534728 ] Steve Loughran commented on SPARK-24746: Mumbai is v4 auth, which isn't directly

[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541915#comment-16541915 ] Steve Loughran commented on SPARK-21962: Yes, assume that: * Htrace goes off t

[jira] [Commented] (SPARK-23683) FileCommitProtocol.instantiate to require 3-arg constructor for dynamic partition overwrite

[ https://issues.apache.org/jira/browse/SPARK-23683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558660#comment-16558660 ] Steve Loughran commented on SPARK-23683: If it's a regression, you could argue f

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572675#comment-16572675 ] Steve Loughran commented on SPARK-22634: If nothing else is using it, correct. A

[jira] [Commented] (SPARK-23050) Structured Streaming with S3 file source duplicates data because of eventual consistency.

[ https://issues.apache.org/jira/browse/SPARK-23050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574276#comment-16574276 ] Steve Loughran commented on SPARK-23050: bq. Is there any way we can avoid happe

[jira] [Commented] (SPARK-22236) CSV I/O: does not respect RFC 4180

[ https://issues.apache.org/jira/browse/SPARK-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575629#comment-16575629 ] Steve Loughran commented on SPARK-22236: I wouldn't recommend changing multiline

[jira] [Updated] (SPARK-23654) Cut jets3t and bouncy castle as dependencies of spark-core

[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23654: --- Summary: Cut jets3t and bouncy castle as dependencies of spark-core (was: Cut jets3t as a d

[jira] [Updated] (SPARK-23654) Cut jets3t as a dependency of spark-core

[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23654: --- Summary: Cut jets3t as a dependency of spark-core (was: Cut jets3t and bouncy castle as dep

[jira] [Created] (SPARK-25111) increment kinesis client/producer lib versions & aws-sdk to match

Steve Loughran created SPARK-25111: -- Summary: increment kinesis client/producer lib versions & aws-sdk to match Key: SPARK-25111 URL: https://issues.apache.org/jira/browse/SPARK-25111 Project: Spark

[jira] [Commented] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581658#comment-16581658 ] Steve Loughran commented on SPARK-24787: yes,, hsync updating the file length is

[jira] [Commented] (SPARK-25111) increment kinesis client/producer lib versions & aws-sdk to match

[ https://issues.apache.org/jira/browse/SPARK-25111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581663#comment-16581663 ] Steve Loughran commented on SPARK-25111: FWIW, it'd be interesting to do a follo

[jira] [Commented] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582911#comment-16582911 ] Steve Loughran commented on SPARK-24771: Linking to the previous PR, as that's g

[jira] [Commented] (SPARK-22236) CSV I/O: does not respect RFC 4180

[ https://issues.apache.org/jira/browse/SPARK-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582930#comment-16582930 ] Steve Loughran commented on SPARK-22236: bq. One can always repartition after re

[jira] [Commented] (SPARK-22236) CSV I/O: does not respect RFC 4180

[ https://issues.apache.org/jira/browse/SPARK-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582933#comment-16582933 ] Steve Loughran commented on SPARK-22236: bq. But is the implication that we can

[jira] [Commented] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583077#comment-16583077 ] Steve Loughran commented on SPARK-24771: All the wire stuff (e.g. to HDFS is pro

[jira] [Commented] (SPARK-25155) Streaming from storage doesn't work when no directories exists

[ https://issues.apache.org/jira/browse/SPARK-25155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16585438#comment-16585438 ] Steve Loughran commented on SPARK-25155: >From SPARK-17159 I have a more cloud-o

[jira] [Created] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

Steve Loughran created SPARK-25180: -- Summary: Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails Key: SPARK-25180 URL: https://issues.apache.org/jira/browse/SPARK-25180

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588001#comment-16588001 ] Steve Loughran commented on SPARK-25180: code snippet was some trivial CSV => OR

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588005#comment-16588005 ] Steve Loughran commented on SPARK-25180: Stack {code} scala> text("hello all!")

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588012#comment-16588012 ] Steve Loughran commented on SPARK-25180: Netty converts the UnknownHostException

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588035#comment-16588035 ] Steve Loughran commented on SPARK-25180: FWIW, there was no in-progress data at

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588053#comment-16588053 ] Steve Loughran commented on SPARK-6305: --- 1. exclusion of log4j 1.x you can only sa

[jira] [Created] (SPARK-25183) Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

Steve Loughran created SPARK-25183: -- Summary: Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise Key: SPARK-25183 URL: https://issues.apache.org/jira/browse/SPA

[jira] [Commented] (SPARK-25126) avoid creating OrcFile.Reader for all orc files

[ https://issues.apache.org/jira/browse/SPARK-25126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589326#comment-16589326 ] Steve Loughran commented on SPARK-25126: + [~dongjoon] > avoid creating OrcFile

[jira] [Commented] (SPARK-20799) Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595633#comment-16595633 ] Steve Loughran commented on SPARK-20799: [~jzijlstra] yes, the final listing. N

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596404#comment-16596404 ] Steve Loughran commented on SPARK-6305: --- bq. Could be possible that nobody is swapp

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596566#comment-16596566 ] Steve Loughran commented on SPARK-25180: Reviewing a bit more, I think the root

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449857#comment-16449857 ] Steve Loughran commented on SPARK-18673: It's a big hive patch, but most of it is

[jira] [Commented] (SPARK-24000) S3A: Create Table should fail on invalid AK/SK

[ https://issues.apache.org/jira/browse/SPARK-24000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450024#comment-16450024 ] Steve Loughran commented on SPARK-24000: We could consider whether or not to rais

[jira] [Updated] (SPARK-23654) Cut jets3t as a dependency of spark-core

[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23654: --- Summary: Cut jets3t as a dependency of spark-core (was: Cut jets3t as a dependency of spark-

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452196#comment-16452196 ] Steve Loughran commented on SPARK-18673: HIVE-16081 commit 93db527f47 contains th

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452215#comment-16452215 ] Steve Loughran commented on SPARK-18673: looking @ our local commit logs, the HDP

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453825#comment-16453825 ] Steve Loughran commented on SPARK-13446: [~Tavis]: can you paste in the stack you

[jira] [Commented] (SPARK-23151) Provide a distribution of Spark with Hadoop 3.0

[ https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453991#comment-16453991 ] Steve Loughran commented on SPARK-23151: well, there's "have everything work on H

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465917#comment-16465917 ] Steve Loughran commented on SPARK-23977: It will need the hadoop-aws module and d

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465925#comment-16465925 ] Steve Loughran commented on SPARK-18673: Josh Rosen added some changes, particula

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465950#comment-16465950 ] Steve Loughran commented on SPARK-18673: Good Q, [~Bidek]. That SPARK-23807 POM f

[jira] [Created] (SPARK-24280) Speed up indexing of files in object stores by using listFiles(path, recursive=true)

Steve Loughran created SPARK-24280: -- Summary: Speed up indexing of files in object stores by using listFiles(path, recursive=true) Key: SPARK-24280 URL: https://issues.apache.org/jira/browse/SPARK-24280

[jira] [Commented] (SPARK-23681) Switch OrcFileFormat to newer hadoop.mapreduce output classes

[ https://issues.apache.org/jira/browse/SPARK-23681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475790#comment-16475790 ] Steve Loughran commented on SPARK-23681: sorry, been offline * yes, cut the versi

[jira] [Commented] (SPARK-19790) OutputCommitCoordinator should not allow another task to commit after an ExecutorFailure

[ https://issues.apache.org/jira/browse/SPARK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482512#comment-16482512 ] Steve Loughran commented on SPARK-19790: Update on this, having spent lots of tim

[jira] [Commented] (SPARK-24271) sc.hadoopConfigurations can not be overwritten in the same spark context

[ https://issues.apache.org/jira/browse/SPARK-24271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16490474#comment-16490474 ] Steve Loughran commented on SPARK-24271: Disabling the s3 cache can be pretty ine

[jira] [Updated] (SPARK-24273) Failure while using .checkpoint method to private S3 store via S3A connector

[ https://issues.apache.org/jira/browse/SPARK-24273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24273: --- Summary: Failure while using .checkpoint method to private S3 store via S3A connector (was:

[jira] [Updated] (SPARK-24273) Failure while using .checkpoint method to private S3 store via S3A connector

[ https://issues.apache.org/jira/browse/SPARK-24273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24273: --- Description: We are getting following error: {code} com.amazonaws.services.s3.model.AmazonS3

[jira] [Commented] (SPARK-24273) Failure while using .checkpoint method to private S3 store via S3A connector

[ https://issues.apache.org/jira/browse/SPARK-24273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16492159#comment-16492159 ] Steve Loughran commented on SPARK-24273: Of course, there's no need to send range

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497928#comment-16497928 ] Steve Loughran commented on SPARK-18673: Jerry, I list up the other commits I'd

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500560#comment-16500560 ] Steve Loughran commented on SPARK-20202: I think you could split things into two

[jira] [Created] (SPARK-24470) RestSubmissionClient to be robust against 404 & non json responses

Steve Loughran created SPARK-24470: -- Summary: RestSubmissionClient to be robust against 404 & non json responses Key: SPARK-24470 URL: https://issues.apache.org/jira/browse/SPARK-24470 Project: Spark

[jira] [Commented] (SPARK-24470) RestSubmissionClient to be robust against 404 & non json responses

[ https://issues.apache.org/jira/browse/SPARK-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502177#comment-16502177 ] Steve Loughran commented on SPARK-24470: stack from the issue {code} Running Spa

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24476: --- Component/s: (was: Spark Core) Structured Streaming > java.net.SocketTi

[jira] [Commented] (SPARK-24476) java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0

[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506002#comment-16506002 ] Steve Loughran commented on SPARK-24476: Switch from s3n to the s3a connector, s

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24476: --- Summary: java.net.SocketTimeoutException: Read timed out under jets3t while running the Spar

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24476: --- Priority: Minor (was: Major) > java.net.SocketTimeoutException: Read timed out under jets3t

[jira] [Updated] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-24492: --- Summary: Endless attempted task when TaskCommitDenied exception writing to S3A (was: Endles

[jira] [Commented] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506009#comment-16506009 ] Steve Loughran commented on SPARK-24492: the retry problem looks like something

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506021#comment-16506021 ] Steve Loughran commented on SPARK-23534: [~jerryshao] is that the HDFS token ide

<    1   2   3   4   5   6   7   8   9   >