[jira] [Created] (SPARK-27343) Use ConfigEntry for hardcoded configs for spark-sql-kafka

hehuiyuan created SPARK-27343: - Summary: Use ConfigEntry for hardcoded configs for spark-sql-kafka Key: SPARK-27343 URL: https://issues.apache.org/jira/browse/SPARK-27343 Project: Spark Issue

[jira] [Created] (SPARK-27342) Optimize limit 0 queries

Aayushmaan Jain created SPARK-27342: --- Summary: Optimize limit 0 queries Key: SPARK-27342 URL: https://issues.apache.org/jira/browse/SPARK-27342 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27341) dead lock between TaskMemoryManager and UnsafeExternalSorter

Wenchen Fan created SPARK-27341: --- Summary: dead lock between TaskMemoryManager and UnsafeExternalSorter Key: SPARK-27341 URL: https://issues.apache.org/jira/browse/SPARK-27341 Project: Spark

[jira] [Resolved] (SPARK-26224) Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe.

[ https://issues.apache.org/jira/browse/SPARK-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26224. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23285

[jira] [Assigned] (SPARK-26224) Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe.

[ https://issues.apache.org/jira/browse/SPARK-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26224: Assignee: Marco Gaido > Results in stackOverFlowError when trying to add 3000 new

[jira] [Commented] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807406#comment-16807406 ] Sandeep Katta commented on SPARK-27337: ---  I will test this scenario and if required will raise PR

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807400#comment-16807400 ] Lantao Jin commented on SPARK-27216: [~irashid] Not yet. It's petty challenging to find out it

[jira] [Updated] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

[ https://issues.apache.org/jira/browse/SPARK-27340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang updated SPARK-27340: Description: When we use data api to write a structured streaming query job we usually specify a

[jira] [Updated] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

[ https://issues.apache.org/jira/browse/SPARK-27340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang updated SPARK-27340: Description: When we use data api to write a structured streaming query job we usually specify a

[jira] [Created] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

Kevin Zhang created SPARK-27340: --- Summary: Alias on TimeWIndow expression may cause watermark metadata lost Key: SPARK-27340 URL: https://issues.apache.org/jira/browse/SPARK-27340 Project: Spark

[jira] [Assigned] (SPARK-25496) Deprecate from_utc_timestamp and to_utc_timestamp

[ https://issues.apache.org/jira/browse/SPARK-25496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25496: --- Assignee: Maxim Gekk > Deprecate from_utc_timestamp and to_utc_timestamp >

[jira] [Resolved] (SPARK-25496) Deprecate from_utc_timestamp and to_utc_timestamp

[ https://issues.apache.org/jira/browse/SPARK-25496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25496. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24195

[jira] [Resolved] (SPARK-27325) Support implicit encoders for LocalDate and Instant

[ https://issues.apache.org/jira/browse/SPARK-27325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27325. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24249

[jira] [Assigned] (SPARK-27325) Support implicit encoders for LocalDate and Instant

[ https://issues.apache.org/jira/browse/SPARK-27325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27325: --- Assignee: Maxim Gekk > Support implicit encoders for LocalDate and Instant >

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807316#comment-16807316 ] Hyukjin Kwon commented on SPARK-27330: -- (I think {{close()}} should be idempotent and documented as

[jira] [Updated] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27330: - Description: in cases where a micro batch is being killed (interrupted), not during actual

[jira] [Comment Edited] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806654#comment-16806654 ] Hyukjin Kwon edited comment on SPARK-27330 at 4/2/19 12:27 AM: --- I have

[jira] [Commented] (SPARK-27335) cannot collect() from Correlation.corr

[ https://issues.apache.org/jira/browse/SPARK-27335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807312#comment-16807312 ] Hyukjin Kwon commented on SPARK-27335: -- Can you show how to reproduce from scratch? Seems I can't

[jira] [Commented] (SPARK-27339) Decimal up cast to higher scale fails while reading parquet to Dataset

[ https://issues.apache.org/jira/browse/SPARK-27339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807307#comment-16807307 ] Hyukjin Kwon commented on SPARK-27339: -- [~wrschneider99], you don't have to clone it. just leave

[jira] [Commented] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807245#comment-16807245 ] shane knapp commented on SPARK-25079: - UPDATE! i successfully created and extensively tested a new

[jira] [Updated] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-25079: Summary: [PYTHON] upgrade python 3.4 -> 3.6 (was: [PYTHON] upgrade python 3.4 -> 3.5) >

[jira] [Updated] (SPARK-27339) Decimal up cast to higher scale fails while reading parquet to Dataset

[ https://issues.apache.org/jira/browse/SPARK-27339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Schneider updated SPARK-27339: --- Affects Version/s: 2.4.0 I am cloning/reopening this issue - the argument for why this

[jira] [Created] (SPARK-27339) CLONE - Decimal up cast to higher scale fails while reading parquet to Dataset

Bill Schneider created SPARK-27339: -- Summary: CLONE - Decimal up cast to higher scale fails while reading parquet to Dataset Key: SPARK-27339 URL: https://issues.apache.org/jira/browse/SPARK-27339

[jira] [Updated] (SPARK-27339) Decimal up cast to higher scale fails while reading parquet to Dataset

[ https://issues.apache.org/jira/browse/SPARK-27339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Schneider updated SPARK-27339: --- Summary: Decimal up cast to higher scale fails while reading parquet to Dataset (was:

[jira] [Resolved] (SPARK-27329) Pruning nested field in map of map key and value from object serializers

[ https://issues.apache.org/jira/browse/SPARK-27329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27329. --- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 3.0.0 This is

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807181#comment-16807181 ] Imran Rashid commented on SPARK-27216: -- Any chance you know what the issue in roaring bitmap is?

[jira] [Comment Edited] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807121#comment-16807121 ] Russell Spitzer edited comment on SPARK-25003 at 4/1/19 7:48 PM: - There

[jira] [Commented] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807121#comment-16807121 ] Russell Spitzer commented on SPARK-25003: - There was no interest in putting in OSS 2.4, but I

[jira] [Updated] (SPARK-12611) test_infer_schema_to_local depended on old handling of missing value in row

[ https://issues.apache.org/jira/browse/SPARK-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Andrunas updated SPARK-12611: -- Attachment: (was: Network Management Downloads.zip) > test_infer_schema_to_local depended

[jira] [Commented] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807073#comment-16807073 ] Sunitha Kambhampati commented on SPARK-25003: - Hi [~hyukjin.kwon], [~rspitzer],   The

[jira] [Resolved] (SPARK-9792) PySpark DenseMatrix, SparseMatrix should override __eq__

[ https://issues.apache.org/jira/browse/SPARK-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-9792. Resolution: Fixed Fix Version/s: 3.0.0 > PySpark DenseMatrix, SparseMatrix should override __eq__ >

[jira] [Updated] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoo Ganesh updated SPARK-27337: - Affects Version/s: (was: 2.4.0) > QueryExecutionListener never cleans up listeners from the

[jira] [Commented] (SPARK-27338) Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator

[ https://issues.apache.org/jira/browse/SPARK-27338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806946#comment-16806946 ] Venkata krishnan Sowrirajan commented on SPARK-27338: - We found this issue in one of

[jira] [Created] (SPARK-27338) Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator

Venkata krishnan Sowrirajan created SPARK-27338: --- Summary: Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator Key: SPARK-27338 URL:

[jira] [Commented] (SPARK-27336) Incorrect DataSet.summary() result

[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806947#comment-16806947 ] Chakravarthi commented on SPARK-27336: -- I'm checking this issue. > Incorrect DataSet.summary()

[jira] [Created] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

Vinoo Ganesh created SPARK-27337: Summary: QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed Key: SPARK-27337 URL: https://issues.apache.org/jira/browse/SPARK-27337

[jira] [Updated] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoo Ganesh updated SPARK-27337: - Attachment: image001-1.png > QueryExecutionListener never cleans up listeners from the bus

[jira] [Resolved] (SPARK-27278) Optimize GetMapValue when the map is a foldable and the key is not

[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27278. --- Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 3.0.0 This is

[jira] [Updated] (SPARK-27278) Optimize GetMapValue when the map is a foldable and the key is not

[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27278: -- Summary: Optimize GetMapValue when the map is a foldable and the key is not (was:

[jira] [Updated] (SPARK-27278) SimplifyExtractValueOps doesn't optimise Literal map access to CASE WHEN ... END

[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27278: -- Labels: (was: Optimization) > SimplifyExtractValueOps doesn't optimise Literal map access

[jira] [Updated] (SPARK-27278) SimplifyExtractValueOps doesn't optimise Literal map access to CASE WHEN ... END

[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27278: -- Issue Type: Improvement (was: Bug) > SimplifyExtractValueOps doesn't optimise Literal map

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-27216: --- Summary: Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue (was: Upgrade

[jira] [Assigned] (SPARK-26812) PushProjectionThroughUnion nullability issue

[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26812: --- Assignee: Marco Gaido > PushProjectionThroughUnion nullability issue >

[jira] [Resolved] (SPARK-26812) PushProjectionThroughUnion nullability issue

[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26812. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23726

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806841#comment-16806841 ] Lantao Jin commented on SPARK-27216: Thanks for reporting this [~aallen]. It's a bug of

[jira] [Commented] (SPARK-27298) Dataset except operation gives different results(dataset count) on Spark 2.3.0 Windows and Spark 2.3.0 Linux environment

[ https://issues.apache.org/jira/browse/SPARK-27298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806766#comment-16806766 ] Sean Owen commented on SPARK-27298: --- I take it back, that's not the issue. There are a few odd things

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-27216: --- Description: HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-27216: --- Summary: Upgrade RoaringBitmap to 0.7.45 (was: Kryo serialization with RoaringBitmap) > Upgrade

[jira] [Updated] (SPARK-27336) Incorrect DataSet.summary() result

[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27336: --- Attachment: test.csv > Incorrect DataSet.summary() result >

[jira] [Created] (SPARK-27336) Incorrect DataSet.summary() result

Gengliang Wang created SPARK-27336: -- Summary: Incorrect DataSet.summary() result Key: SPARK-27336 URL: https://issues.apache.org/jira/browse/SPARK-27336 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27336) Incorrect DataSet.summary() result

[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27336: --- Issue Type: Bug (was: Improvement) > Incorrect DataSet.summary() result >

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806659#comment-16806659 ] Gabor Somogyi commented on SPARK-27330: --- Now I see the point and makes sense from my perspective.

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806654#comment-16806654 ] Eyal Zituny commented on SPARK-27330: - I have updated the description, eventually i have tasks which

[jira] [Updated] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eyal Zituny updated SPARK-27330: Description: in cases where a micro batch is being killed (interrupted), not during actual

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806638#comment-16806638 ] Gabor Somogyi commented on SPARK-27330: --- If processing throws any kind of exception then

[jira] [Comment Edited] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806554#comment-16806554 ] Sujith Chacko edited comment on SPARK-2647 at 4/1/19 9:32 AM: -- [~joshrosen]

[jira] [Comment Edited] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806554#comment-16806554 ] Sujith Chacko edited comment on SPARK-2647 at 4/1/19 9:30 AM: -- [~joshrosen]

[jira] [Commented] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806554#comment-16806554 ] Sujith Chacko commented on SPARK-2647: -- [~joshrosen] - May i know the Jira is where we already

[jira] [Comment Edited] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806554#comment-16806554 ] Sujith Chacko edited comment on SPARK-2647 at 4/1/19 9:30 AM: -- [~joshrosen]

[jira] [Created] (SPARK-27335) cannot collect() from Correlation.corr

Natalino Busa created SPARK-27335: - Summary: cannot collect() from Correlation.corr Key: SPARK-27335 URL: https://issues.apache.org/jira/browse/SPARK-27335 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27208) RestSubmissionClient only supports http

[ https://issues.apache.org/jira/browse/SPARK-27208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806495#comment-16806495 ] Jorge Machado commented on SPARK-27208: --- Any help here please ? I really think this is broken .

[jira] [Updated] (SPARK-27334) Support specify scheduler name for executor pods when submit

[ https://issues.apache.org/jira/browse/SPARK-27334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TommyLike updated SPARK-27334: -- Description: Currently, there are some external schedulers which bring a lot a great value into

[jira] [Created] (SPARK-27334) Support specify scheduler name for executor pods when submit

TommyLike created SPARK-27334: - Summary: Support specify scheduler name for executor pods when submit Key: SPARK-27334 URL: https://issues.apache.org/jira/browse/SPARK-27334 Project: Spark

[jira] [Commented] (SPARK-27284) Spark Standalone aggregated logs in 1 file per appid (ala yarn -logs -applicationId)

[ https://issues.apache.org/jira/browse/SPARK-27284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806462#comment-16806462 ] t oo commented on SPARK-27284: -- [~srowen] in yarn mode if you want all logs for a single spark app you can

[jira] [Commented] (SPARK-27298) Dataset except operation gives different results(dataset count) on Spark 2.3.0 Windows and Spark 2.3.0 Linux environment

[ https://issues.apache.org/jira/browse/SPARK-27298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806415#comment-16806415 ] Mahima Khatri commented on SPARK-27298: --- [~srowen] , There are no newline issues for sure.Since