[jira] [Created] (SPARK-39531) Show "Name" in Spark master UI for drivers

Jack Hu created SPARK-39531: --- Summary: Show "Name" in Spark master UI for drivers Key: SPARK-39531 URL: https://issues.apache.org/jira/browse/SPARK-39531 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-36672) Using value from map in grouping sets result org.apache.spark.sql.AnalysisException

Jack Hu created SPARK-36672: --- Summary: Using value from map in grouping sets result org.apache.spark.sql.AnalysisException Key: SPARK-36672 URL: https://issues.apache.org/jira/browse/SPARK-36672 Project:

[jira] [Updated] (SPARK-36252) Add log files rolling policy for driver running in cluster mode with spark standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-36252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-36252: Description: For a long running driver in cluster mode, there is no rolling policy, the log

[jira] [Created] (SPARK-36252) Add log files rolling policy for driver running in cluster mode with spark standalone cluster

Jack Hu created SPARK-36252: --- Summary: Add log files rolling policy for driver running in cluster mode with spark standalone cluster Key: SPARK-36252 URL: https://issues.apache.org/jira/browse/SPARK-36252

[jira] [Comment Edited] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

[ https://issues.apache.org/jira/browse/SPARK-35027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372425#comment-17372425 ] Jack Hu edited comment on SPARK-35027 at 7/1/21, 6:42 AM: -- Of course, the

[jira] [Comment Edited] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

[ https://issues.apache.org/jira/browse/SPARK-35027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372425#comment-17372425 ] Jack Hu edited comment on SPARK-35027 at 7/1/21, 6:42 AM: -- Of course, the

[jira] [Comment Edited] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

[ https://issues.apache.org/jira/browse/SPARK-35027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372425#comment-17372425 ] Jack Hu edited comment on SPARK-35027 at 7/1/21, 6:40 AM: -- Of course, the

[jira] [Comment Edited] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

[ https://issues.apache.org/jira/browse/SPARK-35027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372425#comment-17372425 ] Jack Hu edited comment on SPARK-35027 at 7/1/21, 6:39 AM: -- Of course, the

[jira] [Commented] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

[ https://issues.apache.org/jira/browse/SPARK-35027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372425#comment-17372425 ] Jack Hu commented on SPARK-35027: - Of course, the "stop" in FileAppender does nothing but set a flag. 

[jira] [Updated] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

[ https://issues.apache.org/jira/browse/SPARK-35027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-35027: Description: In Spark Cluster, the ExecutorRunner uses FileAppender  to redirect the stdout/stderr of

[jira] [Created] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

Jack Hu created SPARK-35027: --- Summary: Close the inputStream in FileAppender when writing the logs failure Key: SPARK-35027 URL: https://issues.apache.org/jira/browse/SPARK-35027 Project: Spark

[jira] [Commented] (SPARK-21480) Memory leak in org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeNoResult

[ https://issues.apache.org/jira/browse/SPARK-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094351#comment-16094351 ] Jack Hu commented on SPARK-21480: - The issue seems resovled in latest HIVE:

[jira] [Created] (SPARK-21480) Memory leak in org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeNoResult

Jack Hu created SPARK-21480: --- Summary: Memory leak in org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeNoResult Key: SPARK-21480 URL: https://issues.apache.org/jira/browse/SPARK-21480 Project:

[jira] [Updated] (SPARK-16413) The JDBC UI does not show the job id when spark.sql.thriftServer.incrementalCollect=true

[ https://issues.apache.org/jira/browse/SPARK-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-16413: Attachment: no_job_id.png ThriftServer2 does not show the "job id" > The JDBC UI does not show the job id

[jira] [Created] (SPARK-16413) The JDBC UI does not show the job id when spark.sql.thriftServer.incrementalCollect=true

Jack Hu created SPARK-16413: --- Summary: The JDBC UI does not show the job id when spark.sql.thriftServer.incrementalCollect=true Key: SPARK-16413 URL: https://issues.apache.org/jira/browse/SPARK-16413

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116545#comment-15116545 ] Jack Hu commented on SPARK-6847: Hi [~zsxwing] Even user does not implicit do checkpoint after the

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114722#comment-15114722 ] Jack Hu commented on SPARK-6847: Test on latest 1.6 branch (f913f7e [SPARK-12120][PYSPARK] Improve

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111849#comment-15111849 ] Jack Hu commented on SPARK-6847: Hi [~zsxwing] I just test a simple case with 1.6, it still exists:

[jira] [Created] (SPARK-11749) Duplicate creating the RDD in file stream when recovering from checkpoint data

Jack Hu created SPARK-11749: --- Summary: Duplicate creating the RDD in file stream when recovering from checkpoint data Key: SPARK-11749 URL: https://issues.apache.org/jira/browse/SPARK-11749 Project: Spark

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951593#comment-14951593 ] Jack Hu commented on SPARK-6847: Hi [~glyton.camilleri] You can check whether there are two dstreams in

[jira] [Updated] (SPARK-7624) Task scheduler delay is increasing time over time in spark local mode

[ https://issues.apache.org/jira/browse/SPARK-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-7624: --- Description: I am running a simple spark streaming program with spark 1.3.1 in local mode, it receives json

[jira] [Updated] (SPARK-7624) Task scheduler delay is increasing time over time in spark local mode

[ https://issues.apache.org/jira/browse/SPARK-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-7624: --- Description: I am running a simple spark streaming program with spark 1.3.1 in local mode, it receives json

[jira] [Updated] (SPARK-7624) Task scheduler delay is increasing time over time in spark local mode

[ https://issues.apache.org/jira/browse/SPARK-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-7624: --- Description: I am running a simple spark streaming program with spark 1.3.1 in local mode, it receives json

[jira] [Created] (SPARK-7624) Task scheduler delay is increasing time over time in spark local mode

Jack Hu created SPARK-7624: -- Summary: Task scheduler delay is increasing time over time in spark local mode Key: SPARK-7624 URL: https://issues.apache.org/jira/browse/SPARK-7624 Project: Spark

[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532108#comment-14532108 ] Jack Hu commented on SPARK-7183: Hi, [~sowen] Do we plan to add this to 1.3+? If there is

[jira] [Updated] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-7183: --- Summary: Memory leak in netty shuffle with spark standalone cluster (was: Memory leak with netty shuffle

[jira] [Commented] (SPARK-6844) Memory leak occurs when register temp table with cache table on

[ https://issues.apache.org/jira/browse/SPARK-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508597#comment-14508597 ] Jack Hu commented on SPARK-6844: Hi, [~marmbrus]] I mean 1.3.X, like 1.3.2. The master

[jira] [Commented] (SPARK-6844) Memory leak occurs when register temp table with cache table on

[ https://issues.apache.org/jira/browse/SPARK-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497627#comment-14497627 ] Jack Hu commented on SPARK-6844: Hi, [~marmbrus] Do we have a plan to port this to 1.3.X

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495914#comment-14495914 ] Jack Hu commented on SPARK-6847: I did a little more investigation about this issue, that

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493804#comment-14493804 ] Jack Hu commented on SPARK-6847: Hi, [~sowen] The checkpoint interval can not be turn

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493517#comment-14493517 ] Jack Hu commented on SPARK-6847: Here is the part of the stack (Full stack at:

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491873#comment-14491873 ] Jack Hu commented on SPARK-6847: Hi, [~sowen] I tested more cases: # only change the

[jira] [Comment Edited] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491873#comment-14491873 ] Jack Hu edited comment on SPARK-6847 at 4/13/15 3:34 AM: - Hi,

[jira] [Commented] (SPARK-6180) Error logged into log4j when use the HiveMetastoreCatalog::tableExists

[ https://issues.apache.org/jira/browse/SPARK-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375336#comment-14375336 ] Jack Hu commented on SPARK-6180: This issue has already fixed at:

[jira] [Closed] (SPARK-6180) Error logged into log4j when use the HiveMetastoreCatalog::tableExists

[ https://issues.apache.org/jira/browse/SPARK-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu closed SPARK-6180. -- Fixed Error logged into log4j when use the HiveMetastoreCatalog::tableExists

[jira] [Resolved] (SPARK-6180) Error logged into log4j when use the HiveMetastoreCatalog::tableExists

[ https://issues.apache.org/jira/browse/SPARK-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu resolved SPARK-6180. Resolution: Fixed Fix Version/s: 1.3.1 Fixed in this pull https://github.com/apache/spark/pull/4365

[jira] [Closed] (SPARK-3275) Socket receiver can not recover when the socket server restarted

[ https://issues.apache.org/jira/browse/SPARK-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu closed SPARK-3275. -- Resolution: Fixed Fix Version/s: 1.1.0 Checked in 1.1.0, do not find similar issue. Socket receiver

[jira] [Commented] (SPARK-6061) File source dstream can not include the old file which timestamp is before the system time

[ https://issues.apache.org/jira/browse/SPARK-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348093#comment-14348093 ] Jack Hu commented on SPARK-6061: [~srowen] The issue is: I want to process the old files

[jira] [Commented] (SPARK-6061) File source dstream can not include the old file which timestamp is before the system time

[ https://issues.apache.org/jira/browse/SPARK-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348230#comment-14348230 ] Jack Hu commented on SPARK-6061: [~tianyi] Do you know why the

[jira] [Created] (SPARK-6180) Error logged into log4j when use the HiveMetastoreCatalog::tableExists

Jack Hu created SPARK-6180: -- Summary: Error logged into log4j when use the HiveMetastoreCatalog::tableExists Key: SPARK-6180 URL: https://issues.apache.org/jira/browse/SPARK-6180 Project: Spark

[jira] [Updated] (SPARK-6180) Error logged into log4j when use the HiveMetastoreCatalog::tableExists

[ https://issues.apache.org/jira/browse/SPARK-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-6180: --- Description: When using {{HiveMetastoreCatalog.tableExists}} to check a table that does not exist in hive

[jira] [Created] (SPARK-6061) File source dstream can not include the old file which timestamp is before the system time

Jack Hu created SPARK-6061: -- Summary: File source dstream can not include the old file which timestamp is before the system time Key: SPARK-6061 URL: https://issues.apache.org/jira/browse/SPARK-6061

[jira] [Commented] (SPARK-3276) Provide a API to specify whether the old files need to be ignored in file input text DStream

[ https://issues.apache.org/jira/browse/SPARK-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283662#comment-14283662 ] Jack Hu commented on SPARK-3276: With some cases, the old files (older than current spark

[jira] [Created] (SPARK-3274) Spark Streaming Java API reports java.lang.ClassCastException when calling collectAsMap on JavaPairDStream

Jack Hu created SPARK-3274: -- Summary: Spark Streaming Java API reports java.lang.ClassCastException when calling collectAsMap on JavaPairDStream Key: SPARK-3274 URL: https://issues.apache.org/jira/browse/SPARK-3274

[jira] [Updated] (SPARK-3274) Spark Streaming Java API reports java.lang.ClassCastException when calling collectAsMap on JavaPairDStream

[ https://issues.apache.org/jira/browse/SPARK-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Hu updated SPARK-3274: --- Description: Reproduce code: scontext .socketTextStream(localhost, 1)

[jira] [Created] (SPARK-3275) Socket receiver can not recover when the socket server restarted

Jack Hu created SPARK-3275: -- Summary: Socket receiver can not recover when the socket server restarted Key: SPARK-3275 URL: https://issues.apache.org/jira/browse/SPARK-3275 Project: Spark Issue

[jira] [Created] (SPARK-3276) Provide a API to specify whether the old files need to be ignored in file input text DStream

Jack Hu created SPARK-3276: -- Summary: Provide a API to specify whether the old files need to be ignored in file input text DStream Key: SPARK-3276 URL: https://issues.apache.org/jira/browse/SPARK-3276