[jira] [Commented] (SPARK-36240) Graceful termination of Spark Structured Streaming queries

2021-11-26 Thread yufei ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449458#comment-17449458 ] yufei ding commented on SPARK-36240: This feature is expected to be available in which version of

[jira] [Updated] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

2021-05-03 Thread Yusheng Ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yusheng Ding updated SPARK-35299: - Description: To reproduce: test_table path: s3a://test_bucket/test_table/   df =

[jira] [Updated] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

2021-05-03 Thread Yusheng Ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yusheng Ding updated SPARK-35299: - Description: To reproduce: test_table path: s3a://test_bucket/test_table/   df =

[jira] [Updated] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

2021-05-03 Thread Yusheng Ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yusheng Ding updated SPARK-35299: - Description: To reproduce: test_table path: s3a://test_bucket/test_table/   df =

[jira] [Updated] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

2021-05-03 Thread Yusheng Ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yusheng Ding updated SPARK-35299: - Description: To reproduce: test_table path: s3a://test_bucket/test_table/   df =

[jira] [Created] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

2021-05-03 Thread Yusheng Ding (Jira)
Yusheng Ding created SPARK-35299: Summary: Dataframe overwrite on S3 does not delete old files with S3 object-put to table path Key: SPARK-35299 URL: https://issues.apache.org/jira/browse/SPARK-35299

[jira] [Updated] (SPARK-31933) Found another deadlock in Spark Driver

2020-06-08 Thread Wenning Ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenning Ding updated SPARK-31933: - Description: This is another Spark java-level deadlock we found: it looks similar to 

[jira] [Updated] (SPARK-31933) Found another deadlock in Spark Driver

2020-06-08 Thread Wenning Ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenning Ding updated SPARK-31933: - Description: This is another Spark java-level deadlock we found: it looks similar to 

[jira] [Updated] (SPARK-31933) Found another deadlock in Spark Driver

2020-06-08 Thread Wenning Ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenning Ding updated SPARK-31933: - Description: This is another Spark java-level deadlock we found: it looks similar to 

[jira] [Created] (SPARK-31933) Found another deadlock in Spark Driver

2020-06-08 Thread Wenning Ding (Jira)
Wenning Ding created SPARK-31933: Summary: Found another deadlock in Spark Driver Key: SPARK-31933 URL: https://issues.apache.org/jira/browse/SPARK-31933 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29868) Add a benchmark for Adaptive Execution

2019-11-12 Thread leonard ding (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972974#comment-16972974 ] leonard ding commented on SPARK-29868: -- I will pick it > Add a benchmark for Adaptive Execution >

[jira] [Comment Edited] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-15 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578987#comment-15578987 ] ding edited comment on SPARK-17951 at 10/16/16 12:22 AM: - I tried to call

[jira] [Commented] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-15 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578987#comment-15578987 ] ding commented on SPARK-17951: -- I tried to call rdd.collect which internally called bm.getRemoteBytes in

[jira] [Updated] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-14 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ding updated SPARK-17951: - Description: The following code demonstrates the issue: def main(args: Array[String]): Unit = { val conf =

[jira] [Created] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-14 Thread ding (JIRA)
ding created SPARK-17951: Summary: BlockFetch with multiple threads slows down after spark 1.6 Key: SPARK-17951 URL: https://issues.apache.org/jira/browse/SPARK-17951 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17559) PeriodicGraphCheckpointer did not persist edges as expected in some cases

2016-10-04 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545991#comment-15545991 ] ding commented on SPARK-17559: -- Yes, it's the correct username. Thank you for your reviewing. >

[jira] [Commented] (SPARK-17097) Pregel does not keep vertex state properly; fails to terminate

2016-09-29 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533976#comment-15533976 ] ding commented on SPARK-17097: -- Because diff of case class behaves different with regular class. A case

[jira] [Commented] (SPARK-5484) Pregel should checkpoint periodically to avoid StackOverflowError

2016-09-16 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496857#comment-15496857 ] ding commented on SPARK-5484: - Thank you for your kindly reminder. However as the code is almost ready, I will

[jira] [Created] (SPARK-17559) PeriodicGraphCheckpointer didnot persist edges as expected in some cases

2016-09-15 Thread ding (JIRA)
ding created SPARK-17559: Summary: PeriodicGraphCheckpointer didnot persist edges as expected in some cases Key: SPARK-17559 URL: https://issues.apache.org/jira/browse/SPARK-17559 Project: Spark

[jira] [Commented] (SPARK-5484) Pregel should checkpoint periodically to avoid StackOverflowError

2016-09-15 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495131#comment-15495131 ] ding commented on SPARK-5484: - I will work on the issue if nobody took it. > Pregel should checkpoint

[jira] [Commented] (SPARK-17097) Pregel does not keep vertex state properly; fails to terminate

2016-09-13 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488609#comment-15488609 ] ding commented on SPARK-17097: -- I am afraid the attached sample code fail to terminate with case class is

[jira] [Commented] (SPARK-16071) Not sufficient array size checks to avoid integer overflows in Tungsten

2016-06-21 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341354#comment-15341354 ] ding commented on SPARK-16071: -- OK, I see. Thank you for your clarification. > Not sufficient array size

[jira] [Commented] (SPARK-16071) Not sufficient array size checks to avoid integer overflows in Tungsten

2016-06-21 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341318#comment-15341318 ] ding commented on SPARK-16071: -- The exception raised in different location as one happened in encoder(n =

[jira] [Commented] (SPARK-15562) Temp directory is not deleted after program exit in DataFrameExample

2016-05-26 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303326#comment-15303326 ] ding commented on SPARK-15562: -- OK, I will check if there is similar JIRAs before creating a new one next

[jira] [Created] (SPARK-15562) Temp directory is not deleted after program exit in DataFrameExample

2016-05-26 Thread ding (JIRA)
ding created SPARK-15562: Summary: Temp directory is not deleted after program exit in DataFrameExample Key: SPARK-15562 URL: https://issues.apache.org/jira/browse/SPARK-15562 Project: Spark Issue

[jira] [Created] (SPARK-15172) Warning message should explicitly tell user initial coefficients is ignored if its size doesn't match expected size in LogisticRegression

2016-05-06 Thread ding (JIRA)
ding created SPARK-15172: Summary: Warning message should explicitly tell user initial coefficients is ignored if its size doesn't match expected size in LogisticRegression Key: SPARK-15172 URL:

[jira] [Reopened] (SPARK-14969) Remove unnecessary compute function in LogisticGradient

2016-04-27 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ding reopened SPARK-14969: -- > Remove unnecessary compute function in LogisticGradient >

[jira] [Resolved] (SPARK-14969) Remove unnecessary compute function in LogisticGradient

2016-04-27 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ding resolved SPARK-14969. -- Resolution: Fixed > Remove unnecessary compute function in LogisticGradient >

[jira] [Created] (SPARK-14969) Remove unnecessary compute function in LogisticGradient

2016-04-27 Thread ding (JIRA)
ding created SPARK-14969: Summary: Remove unnecessary compute function in LogisticGradient Key: SPARK-14969 URL: https://issues.apache.org/jira/browse/SPARK-14969 Project: Spark Issue Type:

[jira] [Commented] (SPARK-14889) scala.MatchError: NONE (of class scala.Enumeration$Val) when spark.scheduler.mode=NONE

2016-04-25 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256061#comment-15256061 ] ding commented on SPARK-14889: -- @Jacek Laskowski, do you mind if I take the jira and give a fix to the

[jira] [Created] (SPARK-13378) Add tee method to RDD

2016-02-18 Thread Richard Ding (JIRA)
Richard Ding created SPARK-13378: Summary: Add tee method to RDD Key: SPARK-13378 URL: https://issues.apache.org/jira/browse/SPARK-13378 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-09-09 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738090#comment-14738090 ] ding commented on SPARK-5556: - We have made the spark package and it can be find here

[jira] [Commented] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-08-14 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696677#comment-14696677 ] ding commented on SPARK-5556: - The code can be found

[jira] [Created] (SPARK-5418) Output directory for shuffle should consider left space of each directory set in conf

2015-01-26 Thread ding (JIRA)
ding created SPARK-5418: --- Summary: Output directory for shuffle should consider left space of each directory set in conf Key: SPARK-5418 URL: https://issues.apache.org/jira/browse/SPARK-5418 Project: Spark

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2015-01-24 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290961#comment-14290961 ] ding commented on SPARK-4105: - I hit this error when using pagerank(It cannot be consistent