[jira] [Comment Edited] (SPARK-10912) Improve Spark metrics executor.filesystem

[ https://issues.apache.org/jira/browse/SPARK-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638836#comment-15638836 ] Yongjia Wang edited comment on SPARK-10912 at 11/5/16 7:09 AM: --- s3a and

[jira] [Reopened] (SPARK-10912) Improve Spark metrics executor.filesystem

[ https://issues.apache.org/jira/browse/SPARK-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang reopened SPARK-10912: -- > Improve Spark metrics executor.filesystem > - > >

[jira] [Commented] (SPARK-10912) Improve Spark metrics executor.filesystem

[ https://issues.apache.org/jira/browse/SPARK-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638836#comment-15638836 ] Yongjia Wang commented on SPARK-10912: -- s3a and hdfs are different "schemes" in Spark's

[jira] [Commented] (SPARK-16484) Incremental Cardinality estimation operations with Hyperloglog

[ https://issues.apache.org/jira/browse/SPARK-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422887#comment-15422887 ] Yongjia Wang commented on SPARK-16484: -- Here is my solution using Spark UDAF and UDT

[jira] [Commented] (SPARK-16484) Incremental Cardinality estimation operations with Hyperloglog

[ https://issues.apache.org/jira/browse/SPARK-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371709#comment-15371709 ] Yongjia Wang commented on SPARK-16484: -- Yes, I agree all the building blocks are there and easy

[jira] [Created] (SPARK-16484) Incremental Cardinality estimation operations with Hyperloglog

Yongjia Wang created SPARK-16484: Summary: Incremental Cardinality estimation operations with Hyperloglog Key: SPARK-16484 URL: https://issues.apache.org/jira/browse/SPARK-16484 Project: Spark

[jira] [Commented] (SPARK-11824) WebUI does not render descriptions with 'bad' HTML, throws console error

[ https://issues.apache.org/jira/browse/SPARK-11824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032602#comment-15032602 ] Yongjia Wang commented on SPARK-11824: -- Looks this is the right escaper. StringEscapeUtils.escapeXml

[jira] [Commented] (SPARK-11824) WebUI does not render descriptions with 'bad' HTML, throws console error

[ https://issues.apache.org/jira/browse/SPARK-11824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031199#comment-15031199 ] Yongjia Wang commented on SPARK-11824: -- Yes, this is annoying and not just for CLI. But you can

[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982662#comment-14982662 ] Yongjia Wang commented on SPARK-11413: -- yea, the fix is to update the joda.version number from 2.5

[jira] [Comment Edited] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982755#comment-14982755 ] Yongjia Wang edited comment on SPARK-11413 at 10/30/15 4:02 PM: My last

[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982802#comment-14982802 ] Yongjia Wang commented on SPARK-11413: -- I see. I don't know, is it safe to assume joda-time is

[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982755#comment-14982755 ] Yongjia Wang commented on SPARK-11413: -- My last statement was wrong. It's a problem when using jre

[jira] [Comment Edited] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982755#comment-14982755 ] Yongjia Wang edited comment on SPARK-11413 at 10/30/15 4:02 PM: My last

[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982874#comment-14982874 ] Yongjia Wang commented on SPARK-11413: -- I can follow up with a PR first. The latest joda-time

[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982909#comment-14982909 ] Yongjia Wang commented on SPARK-11413: -- There is no more transitive dependencies from joda-time. So

[jira] [Created] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version

Yongjia Wang created SPARK-11413: Summary: Java 8 build has problem with joda-time and s3 request, should bump joda-time version Key: SPARK-11413 URL: https://issues.apache.org/jira/browse/SPARK-11413

[jira] [Updated] (SPARK-11354) Expose custom log4j to executor page in Spark standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-11354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-11354: - Attachment: custom log4j on executor page.png > Expose custom log4j to executor page in Spark

[jira] [Created] (SPARK-11354) Expose custom log4j to executor page in Spark standalone cluster

Yongjia Wang created SPARK-11354: Summary: Expose custom log4j to executor page in Spark standalone cluster Key: SPARK-11354 URL: https://issues.apache.org/jira/browse/SPARK-11354 Project: Spark

[jira] [Updated] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

[ https://issues.apache.org/jira/browse/SPARK-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-11175: - Description: Spark StreamingContext can register multiple independent Input DStreams (such as

[jira] [Created] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

Yongjia Wang created SPARK-11175: Summary: Concurrent execution of JobSet within a batch in Spark streaming Key: SPARK-11175 URL: https://issues.apache.org/jira/browse/SPARK-11175 Project: Spark

[jira] [Updated] (SPARK-11152) Streaming UI: Input sizes are 0 for makeup batches started from a checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-11152: - Priority: Major (was: Minor) > Streaming UI: Input sizes are 0 for makeup batches started from

[jira] [Updated] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

[ https://issues.apache.org/jira/browse/SPARK-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-11175: - Description: Spark StreamingContext can register multiple independent Input DStreams (such as

[jira] [Updated] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

[ https://issues.apache.org/jira/browse/SPARK-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-11175: - Description: Spark StreamingContext can register multiple independent Input DStreams (such as

[jira] [Commented] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

[ https://issues.apache.org/jira/browse/SPARK-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962774#comment-14962774 ] Yongjia Wang commented on SPARK-11175: -- nice. should have found this. Thank you > Concurrent

[jira] [Closed] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

[ https://issues.apache.org/jira/browse/SPARK-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang closed SPARK-11175. Resolution: Not A Problem > Concurrent execution of JobSet within a batch in Spark streaming >

[jira] [Updated] (SPARK-11152) Streaming UI: Input sizes are 0 for makeup batches started from a checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-11152: - Description: When a streaming job is resumed from a checkpoint at batch time x, and say the

[jira] [Created] (SPARK-11152) Streaming UI: Input sizes are 0 for makeup batches started from a checkpoint

Yongjia Wang created SPARK-11152: Summary: Streaming UI: Input sizes are 0 for makeup batches started from a checkpoint Key: SPARK-11152 URL: https://issues.apache.org/jira/browse/SPARK-11152

[jira] [Updated] (SPARK-11152) Streaming UI: Input sizes are 0 for makeup batches started from a checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-11152: - Description: When a streaming job starts from a checkpoint at batch time x, and say the current

[jira] [Updated] (SPARK-10912) Improve Spark metrics executor.filesystem

[ https://issues.apache.org/jira/browse/SPARK-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-10912: - Attachment: s3a_metrics.patch Adding s3a is fairly straightforward. I guess the reason it's not

[jira] [Updated] (SPARK-10912) Improve Spark metrics executor.filesystem

[ https://issues.apache.org/jira/browse/SPARK-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-10912: - Description: In org.apache.spark.executor.ExecutorSource it has 2 filesystem metrics: "hdfs"

[jira] [Created] (SPARK-10912) Improve Spark metrics executor.filesystem

Yongjia Wang created SPARK-10912: Summary: Improve Spark metrics executor.filesystem Key: SPARK-10912 URL: https://issues.apache.org/jira/browse/SPARK-10912 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5874) How to improve the current ML pipeline API?

[ https://issues.apache.org/jira/browse/SPARK-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940702#comment-14940702 ] Yongjia Wang commented on SPARK-5874: - The functionality about force save/load all pipeline components

[jira] [Comment Edited] (SPARK-5152) Let metrics.properties file take an hdfs:// path

[ https://issues.apache.org/jira/browse/SPARK-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902883#comment-14902883 ] Yongjia Wang edited comment on SPARK-5152 at 9/22/15 6:00 PM: -- I voted for

[jira] [Commented] (SPARK-5152) Let metrics.properties file take an hdfs:// path

[ https://issues.apache.org/jira/browse/SPARK-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902883#comment-14902883 ] Yongjia Wang commented on SPARK-5152: - I voted for this. It enables configuring metrics or log4j

[jira] [Created] (SPARK-3512) yarn-client through socks proxy

Yongjia Wang created SPARK-3512: --- Summary: yarn-client through socks proxy Key: SPARK-3512 URL: https://issues.apache.org/jira/browse/SPARK-3512 Project: Spark Issue Type: Wish

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a