[jira] [Commented] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646658#comment-17646658 ] huangtengfei commented on SPARK-41497: -- I also think that option3/4(include the imp

[jira] [Created] (SPARK-39853) Support stage level schedule for standalone cluster when dynamic allocation is disabled

huangtengfei created SPARK-39853: Summary: Support stage level schedule for standalone cluster when dynamic allocation is disabled Key: SPARK-39853 URL: https://issues.apache.org/jira/browse/SPARK-39853

[jira] [Commented] (SPARK-39062) Add Standalone backend support for Stage Level Scheduling

[ https://issues.apache.org/jira/browse/SPARK-39062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529717#comment-17529717 ] huangtengfei commented on SPARK-39062: -- I am working on this. Thanks [~jiangxb1987]

[jira] [Commented] (SPARK-38471) Use error classes in org.apache.spark.rdd

[ https://issues.apache.org/jira/browse/SPARK-38471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524688#comment-17524688 ] huangtengfei commented on SPARK-38471: -- I am working on this. > Use error classes

[jira] [Commented] (SPARK-38462) Use error classes in org.apache.spark.executor

[ https://issues.apache.org/jira/browse/SPARK-38462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522331#comment-17522331 ] huangtengfei commented on SPARK-38462: -- I am working on this. Thanks [~bozhang] >

[jira] [Commented] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

[ https://issues.apache.org/jira/browse/SPARK-38689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520541#comment-17520541 ] huangtengfei commented on SPARK-38689: -- I am working on this. Thanks [~maxgekk] >

[jira] [Commented] (SPARK-38108) Use error classes in the compilation errors of UDF/UDAF

[ https://issues.apache.org/jira/browse/SPARK-38108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507528#comment-17507528 ] huangtengfei commented on SPARK-38108: -- I am working on this. Thanks [~maxgekk] >

[jira] [Commented] (SPARK-38106) Use error classes in the parsing errors of functions

[ https://issues.apache.org/jira/browse/SPARK-38106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502884#comment-17502884 ] huangtengfei commented on SPARK-38106: -- I am working on this. Thanks [~maxgekk] >

[jira] [Created] (SPARK-38434) Correct semantic of CheckAnalysis.getDataTypesAreCompatibleFn method

huangtengfei created SPARK-38434: Summary: Correct semantic of CheckAnalysis.getDataTypesAreCompatibleFn method Key: SPARK-38434 URL: https://issues.apache.org/jira/browse/SPARK-38434 Project: Spark

[jira] [Commented] (SPARK-38112) Use error classes in the execution errors of date/timestamp handling

[ https://issues.apache.org/jira/browse/SPARK-38112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492514#comment-17492514 ] huangtengfei commented on SPARK-38112: -- I will work on this. Thanks [~maxgekk] > U

[jira] [Commented] (SPARK-38113) Use error classes in the execution errors of pivoting

[ https://issues.apache.org/jira/browse/SPARK-38113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487833#comment-17487833 ] huangtengfei commented on SPARK-38113: -- I will work on this. Thanks [~maxgekk] > U

[jira] [Commented] (SPARK-38105) Use error classes in the parsing errors of joins

[ https://issues.apache.org/jira/browse/SPARK-38105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487619#comment-17487619 ] huangtengfei commented on SPARK-38105: -- I will work on this. Thanks [~maxgekk] > U

[jira] [Commented] (SPARK-37941) Use error classes in the compilation errors of casting

[ https://issues.apache.org/jira/browse/SPARK-37941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477582#comment-17477582 ] huangtengfei commented on SPARK-37941: -- I will work on this. Thanks [~maxgekk] > U

[jira] [Resolved] (SPARK-36954) Fast fail with explicit err msg when calling withWatermark on non-streaming dataset

[ https://issues.apache.org/jira/browse/SPARK-36954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei resolved SPARK-36954. -- Resolution: Not A Problem > Fast fail with explicit err msg when calling withWatermark on non-

[jira] [Updated] (SPARK-36954) Fast fail with explicit err msg when calling withWatermark on non-streaming dataset

[ https://issues.apache.org/jira/browse/SPARK-36954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei updated SPARK-36954: - Description: [Dataset.withWatermark|https://github.com/apache/spark/blob/v3.2.0-rc7/sql/core/src

[jira] [Updated] (SPARK-36954) Fast fail with explicit err msg when calling withWatermark on non-streaming dataset

[ https://issues.apache.org/jira/browse/SPARK-36954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei updated SPARK-36954: - Environment: (was: [Dataset.withWatermark|https://github.com/apache/spark/blob/v3.2.0-rc7/sq

[jira] [Created] (SPARK-36954) Fast fail with explicit err msg when calling withWatermark on non-streaming dataset

huangtengfei created SPARK-36954: Summary: Fast fail with explicit err msg when calling withWatermark on non-streaming dataset Key: SPARK-36954 URL: https://issues.apache.org/jira/browse/SPARK-36954 P

[jira] [Comment Edited] (SPARK-36658) Expose executionId to QueryExecutionListener

[ https://issues.apache.org/jira/browse/SPARK-36658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409210#comment-17409210 ] huangtengfei edited comment on SPARK-36658 at 9/3/21, 2:36 AM: ---

[jira] [Commented] (SPARK-36658) Expose executionId to QueryExecutionListener

[ https://issues.apache.org/jira/browse/SPARK-36658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409228#comment-17409228 ] huangtengfei commented on SPARK-36658: -- Will create a RP for this. > Expose execut

[jira] [Updated] (SPARK-36658) Expose executionId to QueryExecutionListener

[ https://issues.apache.org/jira/browse/SPARK-36658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei updated SPARK-36658: - Description: Now in [QueryExecutionListener|https://github.com/apache/spark/blob/v3.2.0-rc2/sql

[jira] [Commented] (SPARK-36658) Expose executionId to QueryExecutionListener

[ https://issues.apache.org/jira/browse/SPARK-36658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409210#comment-17409210 ] huangtengfei commented on SPARK-36658: -- cc [~cloud_fan] could you share any thought

[jira] [Created] (SPARK-36658) Expose executionId to QueryExecutionListener

huangtengfei created SPARK-36658: Summary: Expose executionId to QueryExecutionListener Key: SPARK-36658 URL: https://issues.apache.org/jira/browse/SPARK-36658 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-35411) Essential information missing in TreeNode json string

[ https://issues.apache.org/jira/browse/SPARK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei updated SPARK-35411: - Description: TreeNode can be serialized to json string with the method toJSON() or prettyJson()

[jira] [Updated] (SPARK-35411) Essential information missing in TreeNode json string

[ https://issues.apache.org/jira/browse/SPARK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei updated SPARK-35411: - Description: TreeNode can be serialized to json string with the method toJSON() or prettyJson()

[jira] [Comment Edited] (SPARK-35411) Essential information missing in TreeNode json string

[ https://issues.apache.org/jira/browse/SPARK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345060#comment-17345060 ] huangtengfei edited comment on SPARK-35411 at 5/15/21, 3:10 PM: --

[jira] [Comment Edited] (SPARK-35411) Essential information missing in TreeNode json string

[ https://issues.apache.org/jira/browse/SPARK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345060#comment-17345060 ] huangtengfei edited comment on SPARK-35411 at 5/15/21, 3:10 PM: --

[jira] [Commented] (SPARK-35411) Essential information missing in TreeNode json string

[ https://issues.apache.org/jira/browse/SPARK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345060#comment-17345060 ] huangtengfei commented on SPARK-35411: -- Maybe we can write out product objects whic

[jira] [Created] (SPARK-35411) Essential information missing in TreeNode json string

huangtengfei created SPARK-35411: Summary: Essential information missing in TreeNode json string Key: SPARK-35411 URL: https://issues.apache.org/jira/browse/SPARK-35411 Project: Spark Issue T

[jira] [Commented] (SPARK-10816) EventTime based sessionization

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16698433#comment-16698433 ] huangtengfei commented on SPARK-10816: -- Ran the benchmark [~kabhwan] mentioned abov

[jira] [Created] (SPARK-25261) Update configuration.md, correct the default units of spark.driver|executor.memory

huangtengfei created SPARK-25261: Summary: Update configuration.md, correct the default units of spark.driver|executor.memory Key: SPARK-25261 URL: https://issues.apache.org/jira/browse/SPARK-25261 Pr

[jira] [Updated] (SPARK-24351) offsetLog/commitLog purge thresholdBatchId should be computed with current committed epoch but not currentBatchId in CP mode

[ https://issues.apache.org/jira/browse/SPARK-24351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei updated SPARK-24351: - Description: In structured streaming, there is a conf spark.sql.streaming.minBatchesToRetain whi

[jira] [Created] (SPARK-24351) offsetLog/commitLog purge thresholdBatchId should be computed with current committed epoch but not currentBatchId in CP mode

huangtengfei created SPARK-24351: Summary: offsetLog/commitLog purge thresholdBatchId should be computed with current committed epoch but not currentBatchId in CP mode Key: SPARK-24351 URL: https://issues.apache.o

[jira] [Commented] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357952#comment-16357952 ] huangtengfei commented on SPARK-23053: -- the following is a repro case, for clarity 

[jira] [Commented] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353761#comment-16353761 ] huangtengfei commented on SPARK-23053: -- here is the stack trace of exception. java.

[jira] [Comment Edited] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353761#comment-16353761 ] huangtengfei edited comment on SPARK-23053 at 2/6/18 11:48 AM:

[jira] [Updated] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei updated SPARK-23053: - Description: When we run concurrent jobs using the same rdd which is marked to do checkpoint. If

[jira] [Created] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

huangtengfei created SPARK-23053: Summary: taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status Key: SPARK-23053 URL: https://issues.a