[jira] [Commented] (SPARK-51072) CallerContext to set Hadoop cloud client audit context

[ https://issues.apache.org/jira/browse/SPARK-51072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17932644#comment-17932644 ] Steve Loughran commented on SPARK-51072: thanks! > CallerContext to set Hadoop

[jira] [Commented] (SPARK-50859) Upgrade AWS SDK v2 to 2.25.53

[ https://issues.apache.org/jira/browse/SPARK-50859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17923663#comment-17923663 ] Steve Loughran commented on SPARK-50859: fyi, new SDK fixes the inability of the

[jira] [Updated] (SPARK-51072) CallerContext to set Hadoop cloud client audit context

[ https://issues.apache.org/jira/browse/SPARK-51072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-51072: --- Target Version/s: 4.1.0 (was: 4.0.0) > CallerContext to set Hadoop cloud client audit conte

[jira] [Created] (SPARK-51072) CallerContext to set Hadoop cloud client audit context

Steve Loughran created SPARK-51072: -- Summary: CallerContext to set Hadoop cloud client audit context Key: SPARK-51072 URL: https://issues.apache.org/jira/browse/SPARK-51072 Project: Spark Is

[jira] [Commented] (SPARK-49508) Optimized hadoop-aws dependency, aws-java-sdk-bundle jar is too large

[ https://issues.apache.org/jira/browse/SPARK-49508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17891015#comment-17891015 ] Steve Loughran commented on SPARK-49508: > hadoop aws only requires the use of a

[jira] [Commented] (SPARK-48571) Reduce the number of accesses to S3 object storage

[ https://issues.apache.org/jira/browse/SPARK-48571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875510#comment-17875510 ] Steve Loughran commented on SPARK-48571: +that "load a file < fixed length direc

[jira] [Updated] (SPARK-48571) Reduce the number of accesses to S3 object storage

[ https://issues.apache.org/jira/browse/SPARK-48571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-48571: --- Description: If we access a Spark table on an object storage file system with parquet files,

[jira] [Updated] (SPARK-48571) Reduce the number of accesses to S3 object storage

[ https://issues.apache.org/jira/browse/SPARK-48571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-48571: --- Description: If we access a Spark table on an object storage file system with parquet files,

[jira] [Commented] (SPARK-48867) Upgrade the kubernetes-client dependency okhttp version 4.12

[ https://issues.apache.org/jira/browse/SPARK-48867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874854#comment-17874854 ] Steve Loughran commented on SPARK-48867: If you aren't using huawei cloud you ca

[jira] [Commented] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867743#comment-17867743 ] Steve Loughran commented on SPARK-44884: FWIW The new manifest committer, writte

[jira] [Commented] (SPARK-48292) Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

[ https://issues.apache.org/jira/browse/SPARK-48292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866966#comment-17866966 ] Steve Loughran commented on SPARK-48292: what happens if a TA is authorized to c

[jira] [Commented] (SPARK-48571) Reduce the number of accesses to S3 object storage

[ https://issues.apache.org/jira/browse/SPARK-48571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854030#comment-17854030 ] Steve Loughran commented on SPARK-48571: The hadoop openFile() code came with HA

[jira] [Commented] (SPARK-44970) Spark History File Uploads Can Fail on S3

[ https://issues.apache.org/jira/browse/SPARK-44970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850475#comment-17850475 ] Steve Loughran commented on SPARK-44970: correct. file is only saved on close()

[jira] [Commented] (SPARK-47008) Spark to support S3 Express One Zone Storage

[ https://issues.apache.org/jira/browse/SPARK-47008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846014#comment-17846014 ] Steve Loughran commented on SPARK-47008: yes, that looks like it. real PITA this

[jira] [Commented] (SPARK-48123) Provide a constant table schema for querying structured logs

[ https://issues.apache.org/jira/browse/SPARK-48123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844245#comment-17844245 ] Steve Loughran commented on SPARK-48123: this doesn't handle nested stack traces

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829521#comment-17829521 ] Steve Loughran commented on SPARK-38330: [~jpanda] a bit late but your problem i

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822572#comment-17822572 ] Steve Loughran commented on SPARK-41392: expect an official release this week; t

[jira] [Updated] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-41392: --- Priority: Major (was: Minor) > spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in sca

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821694#comment-17821694 ] Steve Loughran commented on SPARK-41392: Hadoop 3.4.0 RC2 exhibits this; spark n

[jira] [Created] (SPARK-47008) Spark to support S3 Express One Zone Storage

Steve Loughran created SPARK-47008: -- Summary: Spark to support S3 Express One Zone Storage Key: SPARK-47008 URL: https://issues.apache.org/jira/browse/SPARK-47008 Project: Spark Issue Type:

[jira] [Commented] (SPARK-45404) Support AWS_ENDPOINT_URL env variable

[ https://issues.apache.org/jira/browse/SPARK-45404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809536#comment-17809536 ] Steve Loughran commented on SPARK-45404: Just saw this while working on SPARK-35

[jira] [Updated] (SPARK-46793) Revert S3A endpoint fixup logic of SPARK-35878

[ https://issues.apache.org/jira/browse/SPARK-46793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-46793: --- Summary: Revert S3A endpoint fixup logic of SPARK-35878 (was: Revert region fixup logic of

[jira] [Created] (SPARK-46793) Revert region fixup logic of SPARK-35878

Steve Loughran created SPARK-46793: -- Summary: Revert region fixup logic of SPARK-35878 Key: SPARK-46793 URL: https://issues.apache.org/jira/browse/SPARK-46793 Project: Spark Issue Type: Sub-

[jira] [Commented] (SPARK-46247) Invalid bucket file error when reading from bucketed table created with PathOutputCommitProtocol

[ https://issues.apache.org/jira/browse/SPARK-46247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808227#comment-17808227 ] Steve Loughran commented on SPARK-46247: why is the file invalid? any more stack

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779460#comment-17779460 ] Steve Loughran commented on SPARK-44124: good document # I think you could cons

[jira] [Commented] (SPARK-38958) Override S3 Client in Spark Write/Read calls

[ https://issues.apache.org/jira/browse/SPARK-38958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763725#comment-17763725 ] Steve Loughran commented on SPARK-38958: [~hershalb] hadoop trunk is now on v2 s

[jira] [Commented] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759063#comment-17759063 ] Steve Loughran commented on SPARK-44884: so using insert overwrite. yes, what ha

[jira] [Commented] (SPARK-44884) Spark doesn't create SUCCESS file when external path is passed

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758167#comment-17758167 ] Steve Loughran commented on SPARK-44884: i'm not trying to replicate it; i have

[jira] [Commented] (SPARK-44884) Spark doesn't create SUCCESS file when external path is passed

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757547#comment-17757547 ] Steve Loughran commented on SPARK-44884: [~dipayandev] i don't think think anyon

[jira] [Commented] (SPARK-38958) Override S3 Client in Spark Write/Read calls

[ https://issues.apache.org/jira/browse/SPARK-38958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757364#comment-17757364 ] Steve Loughran commented on SPARK-38958: [~hershalb] we are about to merge the v

[jira] [Commented] (SPARK-44884) Spark doesn't create SUCCESS file when external path is passed

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757036#comment-17757036 ] Steve Loughran commented on SPARK-44884: this is created in the committer; for h

[jira] [Resolved] (SPARK-44883) Spark insertInto with location GCS bucket root causes NPE

[ https://issues.apache.org/jira/browse/SPARK-44883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-44883. Resolution: Duplicate > Spark insertInto with location GCS bucket root causes NPE > --

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754698#comment-17754698 ] Steve Loughran commented on SPARK-44124: +will need to make sure any classloader

[jira] [Commented] (SPARK-44116) Utilize Hadoop vectorized APIs

[ https://issues.apache.org/jira/browse/SPARK-44116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749266#comment-17749266 ] Steve Loughran commented on SPARK-44116: If this gets into the libraries, you do

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749264#comment-17749264 ] Steve Loughran commented on SPARK-44124: we are soon to move hadoop trunk up to

[jira] [Commented] (SPARK-44042) SPIP: PySpark Test Framework

[ https://issues.apache.org/jira/browse/SPARK-44042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735646#comment-17735646 ] Steve Loughran commented on SPARK-44042: * you can create an independent git rep

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17733454#comment-17733454 ] Steve Loughran commented on SPARK-41599: correct. remember, all the source of ha

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717611#comment-17717611 ] Steve Loughran commented on SPARK-43170: FWIW, using S3 URLs 's3://x/dwm_u

[jira] [Commented] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694969#comment-17694969 ] Steve Loughran commented on SPARK-40034: thanks for the update. I will get that

[jira] [Commented] (SPARK-42537) Remove obsolete/superfluous imports in spark-hadoop-cloud module

[ https://issues.apache.org/jira/browse/SPARK-42537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692617#comment-17692617 ] Steve Loughran commented on SPARK-42537: FYI +[~dannycjones]. I'm getting build

[jira] [Created] (SPARK-42537) Remove obsolete/superfluous imports in spark-hadoop-cloud module

Steve Loughran created SPARK-42537: -- Summary: Remove obsolete/superfluous imports in spark-hadoop-cloud module Key: SPARK-42537 URL: https://issues.apache.org/jira/browse/SPARK-42537 Project: Spark

[jira] [Commented] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678814#comment-17678814 ] Steve Loughran commented on SPARK-40034: Note that these changes aren't sufficie

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678189#comment-17678189 ] Steve Loughran commented on SPARK-41599: well, the challenge there becomes "not

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651672#comment-17651672 ] Steve Loughran commented on SPARK-41599: apps can call  FileSystem.closeAllForUG

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651614#comment-17651614 ] Steve Loughran commented on SPARK-41599: 1. try explicitly disabling the cache f

[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning

[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651384#comment-17651384 ] Steve Loughran commented on SPARK-41551: PR up. PathOutputCommitProtocol stops a

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651201#comment-17651201 ] Steve Loughran commented on SPARK-41599: either the fs is being created by ((Fil

[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning

[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649775#comment-17649775 ] Steve Loughran commented on SPARK-41551: So there's an interesting little "featu

[jira] [Created] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning

Steve Loughran created SPARK-41551: -- Summary: Improve/complete PathOutputCommitProtocol support for dynamic partitioning Key: SPARK-41551 URL: https://issues.apache.org/jira/browse/SPARK-41551 Projec

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643774#comment-17643774 ] Steve Loughran commented on SPARK-41392: may relate to the bouncy castle 1.68 up

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643492#comment-17643492 ] Steve Loughran commented on SPARK-41392: MBP m1 with {code} uname -a Darwin st

[jira] [Created] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

Steve Loughran created SPARK-41392: -- Summary: spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin Key: SPARK-41392 URL: https://issues.apache.org/jira/browse/SPARK-41392 Proje

[jira] [Commented] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17622378#comment-17622378 ] Steve Loughran commented on SPARK-38934: sounds like there is a race condition,

[jira] [Updated] (SPARK-29729) Upgrade ASM to 7.2

[ https://issues.apache.org/jira/browse/SPARK-29729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-29729: --- Description: this patch is required for spark to build with any version of bouncy castle jar

[jira] [Created] (SPARK-40640) SparkHadoopUtil to set origin of hadoop/hive config options

Steve Loughran created SPARK-40640: -- Summary: SparkHadoopUtil to set origin of hadoop/hive config options Key: SPARK-40640 URL: https://issues.apache.org/jira/browse/SPARK-40640 Project: Spark

[jira] [Created] (SPARK-40567) SharedState to redact secrets when propagating them to HadoopConf

Steve Loughran created SPARK-40567: -- Summary: SharedState to redact secrets when propagating them to HadoopConf Key: SPARK-40567 URL: https://issues.apache.org/jira/browse/SPARK-40567 Project: Spark

[jira] [Commented] (SPARK-40286) Load Data from S3 deletes data source file

[ https://issues.apache.org/jira/browse/SPARK-40286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598837#comment-17598837 ] Steve Loughran commented on SPARK-40286: this is EMR. can you repliacate in an A

[jira] [Commented] (SPARK-40287) Load Data using Spark by a single partition moves entire dataset under same location in S3

[ https://issues.apache.org/jira/browse/SPARK-40287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598835#comment-17598835 ] Steve Loughran commented on SPARK-40287: does this happen when # you switch to a

[jira] [Commented] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585275#comment-17585275 ] Steve Loughran commented on SPARK-38934: [~graceee318] try explicitly setting th

[jira] [Reopened] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened SPARK-38934: > Provider TemporaryAWSCredentialsProvider has no credentials > --

[jira] [Commented] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585268#comment-17585268 ] Steve Loughran commented on SPARK-38934: staring at this some more, as there's e

[jira] [Commented] (SPARK-38954) Implement sharing of cloud credentials among driver and executors

[ https://issues.apache.org/jira/browse/SPARK-38954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580711#comment-17580711 ] Steve Loughran commented on SPARK-38954: any plans to put the PR up? i'm curious

[jira] [Commented] (SPARK-38445) Are hadoop committers used in Structured Streaming?

[ https://issues.apache.org/jira/browse/SPARK-38445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580710#comment-17580710 ] Steve Loughran commented on SPARK-38445: SPARK-40039 might address this > Are h

[jira] [Comment Edited] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580707#comment-17580707 ] Steve Loughran edited comment on SPARK-38330 at 8/17/22 9:46 AM: -

[jira] [Comment Edited] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580707#comment-17580707 ] Steve Loughran edited comment on SPARK-38330 at 8/17/22 9:45 AM: -

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580707#comment-17580707 ] Steve Loughran commented on SPARK-38330: remove all jars with cos in the title f

[jira] [Commented] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17578334#comment-17578334 ] Steve Loughran commented on SPARK-40039: doesn't actualy use MPU; if you haven't

[jira] [Updated] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-40034: --- Summary: PathOutputCommitters to work with dynamic partition overwrite (was: PathOutputComm

[jira] [Created] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite -if they support it

Steve Loughran created SPARK-40034: -- Summary: PathOutputCommitters to work with dynamic partition overwrite -if they support it Key: SPARK-40034 URL: https://issues.apache.org/jira/browse/SPARK-40034

[jira] [Commented] (SPARK-39969) Spark AWS SDK and kinesis dependencies lagging hadoop-aws and s3a

[ https://issues.apache.org/jira/browse/SPARK-39969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577389#comment-17577389 ] Steve Loughran commented on SPARK-39969: there's an AWS SDK CVE which is fixed w

[jira] [Commented] (SPARK-39863) Upgrade Hadoop to 3.3.4

[ https://issues.apache.org/jira/browse/SPARK-39863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17574722#comment-17574722 ] Steve Loughran commented on SPARK-39863: probably should follow this with an upg

[jira] [Commented] (SPARK-39969) Spark AWS SDK and kinesis dependencies lagging hadoop-aws and s3a

[ https://issues.apache.org/jira/browse/SPARK-39969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17574720#comment-17574720 ] Steve Loughran commented on SPARK-39969: note: although the latest release fixes

[jira] [Created] (SPARK-39969) Spark AWS SDK and kinesis dependencies lagging hadoop-aws and s3a

Steve Loughran created SPARK-39969: -- Summary: Spark AWS SDK and kinesis dependencies lagging hadoop-aws and s3a Key: SPARK-39969 URL: https://issues.apache.org/jira/browse/SPARK-39969 Project: Spark

[jira] [Updated] (SPARK-39969) Spark AWS SDK and kinesis dependencies lagging hadoop-aws and s3a

[ https://issues.apache.org/jira/browse/SPARK-39969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-39969: --- Priority: Minor (was: Major) > Spark AWS SDK and kinesis dependencies lagging hadoop-aws an

[jira] [Commented] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573653#comment-17573653 ] Steve Loughran commented on SPARK-38934: bq. our system set the provider as WebI

[jira] [Commented] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573652#comment-17573652 ] Steve Loughran commented on SPARK-38934: because its your deployment setup, not

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573141#comment-17573141 ] Steve Loughran commented on SPARK-38330: the hadoop 3.3.4 rC0 will fix this with

[jira] [Commented] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573138#comment-17573138 ] Steve Loughran commented on SPARK-38934: so that's a config problem? not a bug?

[jira] [Resolved] (SPARK-38934) Provider TemporaryAWSCredentialsProvider has no credentials

[ https://issues.apache.org/jira/browse/SPARK-38934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-38934. Resolution: Invalid > Provider TemporaryAWSCredentialsProvider has no credentials > --

[jira] [Commented] (SPARK-38958) Override S3 Client in Spark Write/Read calls

[ https://issues.apache.org/jira/browse/SPARK-38958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573137#comment-17573137 ] Steve Loughran commented on SPARK-38958: #. api is public, but we have changed t

[jira] [Commented] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17570936#comment-17570936 ] Steve Loughran commented on SPARK-33088: i;m playing with this and IOStatistics

[jira] [Commented] (SPARK-29250) Upgrade to Hadoop 3.3.1

[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17553744#comment-17553744 ] Steve Loughran commented on SPARK-29250: use whatever version the spark release

[jira] [Commented] (SPARK-38954) Implement sharing of cloud credentials among driver and executors

[ https://issues.apache.org/jira/browse/SPARK-38954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541002#comment-17541002 ] Steve Loughran commented on SPARK-38954: what is the strategy for having the wor

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17525719#comment-17525719 ] Steve Loughran commented on SPARK-38330: aws sdk does its own thing sometimes, f

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17523891#comment-17523891 ] Steve Loughran commented on SPARK-38330: FWIW I'm not 100% sure this is fixed, a

[jira] [Commented] (SPARK-38445) Are hadoop committers used in Structured Streaming?

[ https://issues.apache.org/jira/browse/SPARK-38445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517556#comment-17517556 ] Steve Loughran commented on SPARK-38445: not suppoorted unless you provide the P

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512458#comment-17512458 ] Steve Loughran commented on SPARK-38652: have you tried running the same suite a

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510830#comment-17510830 ] Steve Loughran commented on SPARK-38330: the hadoop fix is in, but it will take

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507557#comment-17507557 ] Steve Loughran commented on SPARK-38330: sorry about that. try enabling path sty

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504286#comment-17504286 ] Steve Loughran commented on SPARK-38330: this is a hadoop issue -create a Jira t

[jira] [Resolved] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-31911. Fix Version/s: 3.0.1 2.4.7 Resolution: Fixed > Using S3A staging

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504273#comment-17504273 ] Steve Loughran commented on SPARK-31911: I'm going to close as fixed now; the sp

[jira] [Created] (SPARK-38394) build of spark sql against hadoop-3.4.0-snapshot failing with bouncycastle classpath error

Steve Loughran created SPARK-38394: -- Summary: build of spark sql against hadoop-3.4.0-snapshot failing with bouncycastle classpath error Key: SPARK-38394 URL: https://issues.apache.org/jira/browse/SPARK-38394

[jira] [Commented] (SPARK-38115) No spark conf to control the path of _temporary when writing to target filesystem

[ https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17496022#comment-17496022 ] Steve Loughran commented on SPARK-38115: bq. Is there any config as such to stop

[jira] [Commented] (SPARK-38115) No spark conf to control the path of _temporary when writing to target filesystem

[ https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492810#comment-17492810 ] Steve Loughran commented on SPARK-38115: * stop using the classic FileOutputComm

[jira] [Comment Edited] (SPARK-37814) Migrating from log4j 1 to log4j 2

[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488804#comment-17488804 ] Steve Loughran edited comment on SPARK-37814 at 2/8/22, 12:04 PM:

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488804#comment-17488804 ] Steve Loughran commented on SPARK-37814: everyone is aware of the log4j issues,

[jira] [Commented] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485973#comment-17485973 ] Steve Loughran commented on SPARK-37771: [~ivan.sadikov] -any update here? > Ra

[jira] [Commented] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470842#comment-17470842 ] Steve Loughran commented on SPARK-37771: probably related to HADOOP-17372, which

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469178#comment-17469178 ] Steve Loughran commented on SPARK-37814: be good to link to all issues related t

  1   2   3   4   5   6   7   8   9   >