[jira] [Updated] (SPARK-20079) Re registration of AM hangs spark cluster in yarn-client mode

2017-04-03 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-20079: Description: The ExecutorAllocationManager.reset method is called when re-registering AM, which

[jira] [Commented] (SPARK-20079) Re registration of AM hangs spark cluster in yarn-client mode

2017-03-30 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949164#comment-15949164 ] Guoqiang Li commented on SPARK-20079: - I met the problem when doing reliability test. After the

[jira] [Created] (SPARK-20079) Re registration of AM hangs spark cluster in yarn-client mode

2017-03-23 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-20079: --- Summary: Re registration of AM hangs spark cluster in yarn-client mode Key: SPARK-20079 URL: https://issues.apache.org/jira/browse/SPARK-20079 Project: Spark

[jira] [Commented] (SPARK-18890) Do all task serialization in CoarseGrainedExecutorBackend thread (rather than TaskSchedulerImpl)

2017-03-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931115#comment-15931115 ] Guoqiang Li commented on SPARK-18890: - [~kayousterhout] done. > Do all task serialization in

[jira] [Created] (SPARK-19991) FileSegmentManagedBuffer performance improvement.

2017-03-16 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-19991: --- Summary: FileSegmentManagedBuffer performance improvement. Key: SPARK-19991 URL: https://issues.apache.org/jira/browse/SPARK-19991 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19991) FileSegmentManagedBuffer performance improvement.

2017-03-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-19991: Description: When we do not set the value of the configuration items

[jira] [Commented] (SPARK-18890) Do all task serialization in CoarseGrainedExecutorBackend thread (rather than TaskSchedulerImpl)

2017-01-08 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15809212#comment-15809212 ] Guoqiang Li commented on SPARK-18890: - OK, I see. In fact, the most difficult to deal with the

[jira] [Commented] (SPARK-18890) Do all task serialization in CoarseGrainedExecutorBackend thread (rather than TaskSchedulerImpl)

2016-12-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753833#comment-15753833 ] Guoqiang Li commented on SPARK-18890: - [~kayousterhout] I ran a test having 10 million tasks, and did

[jira] [Updated] (SPARK-18383) Utils.isBindCollision does not properly handle all possible address-port collisions when binding

2016-11-09 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-18383: Affects Version/s: 1.6.2 2.0.1 > Utils.isBindCollision does not properly

[jira] [Updated] (SPARK-18383) Utils.isBindCollision does not properly handle all possible address-port collisions when binding

2016-11-09 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-18383: Component/s: Spark Core > Utils.isBindCollision does not properly handle all possible address-port

[jira] [Created] (SPARK-18383) Utils.isBindCollision does not properly handle all possible address-port collisions when binding

2016-11-09 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-18383: --- Summary: Utils.isBindCollision does not properly handle all possible address-port collisions when binding Key: SPARK-18383 URL: https://issues.apache.org/jira/browse/SPARK-18383

[jira] [Updated] (SPARK-18375) Upgrade netty to 4.0.42.Final

2016-11-09 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-18375: Issue Type: Improvement (was: Bug) > Upgrade netty to 4.0.42.Final >

[jira] [Updated] (SPARK-18375) Upgrade netty to 4.0.42.Final

2016-11-08 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-18375: Summary: Upgrade netty to 4.0.42.Final (was: Upgrade netty to 4.0.42) > Upgrade netty to

[jira] [Updated] (SPARK-18375) Upgrade netty to 4.0.42

2016-11-08 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-18375: Summary: Upgrade netty to 4.0.42 (was: Upgrade netty to 4.042) > Upgrade netty to 4.0.42 >

[jira] [Updated] (SPARK-18375) Upgrade netty to 4.042

2016-11-08 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-18375: Description: One of the important changes for 4.0.42.Final is "Support any FileRegion

[jira] [Updated] (SPARK-18375) Upgrade netty to 4.042

2016-11-08 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-18375: Affects Version/s: 1.6.2 > Upgrade netty to 4.042 > -- > >

[jira] [Created] (SPARK-18375) Upgrade netty to 4.042

2016-11-08 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-18375: --- Summary: Upgrade netty to 4.042 Key: SPARK-18375 URL: https://issues.apache.org/jira/browse/SPARK-18375 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-17930) The SerializerInstance instance used when deserializing a TaskResult is not reused

2016-10-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581119#comment-15581119 ] Guoqiang Li commented on SPARK-17930: - TPC-DS 2T data (Parquet) and the SQL(query 2) => {noformat}

[jira] [Commented] (SPARK-17930) The SerializerInstance instance used when deserializing a TaskResult is not reused

2016-10-15 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577506#comment-15577506 ] Guoqiang Li commented on SPARK-17930: - If a stage contains a lot of tasks, eg one million tasks, the

[jira] [Created] (SPARK-17931) taskScheduler has some unneeded serialization

2016-10-13 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-17931: --- Summary: taskScheduler has some unneeded serialization Key: SPARK-17931 URL: https://issues.apache.org/jira/browse/SPARK-17931 Project: Spark Issue Type:

[jira] [Created] (SPARK-17930) The SerializerInstance instance used when deserializing a TaskResult is not reused

2016-10-13 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-17930: --- Summary: The SerializerInstance instance used when deserializing a TaskResult is not reused Key: SPARK-17930 URL: https://issues.apache.org/jira/browse/SPARK-17930

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-09-22 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515042#comment-15515042 ] Guoqiang Li commented on SPARK-6235: ping [~rxin] > Address various 2G limits >

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-09-20 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508670#comment-15508670 ] Guoqiang Li commented on SPARK-6235: [~rxin] Any comments? > Address various 2G limits >

[jira] [Updated] (SPARK-6235) Address various 2G limits

2016-09-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-6235: --- Attachment: (was: SPARK-6235_Design_V0.01.pdf) > Address various 2G limits >

[jira] [Updated] (SPARK-6235) Address various 2G limits

2016-09-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-6235: --- Attachment: SPARK-6235_Design_V0.02.pdf > Address various 2G limits > - > >

[jira] [Updated] (SPARK-6235) Address various 2G limits

2016-08-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-6235: --- Attachment: SPARK-6235_Design_V0.01.pdf Preliminary Design Document. > Address various 2G limits >

[jira] [Commented] (SPARK-17082) Replace ByteBuffer with ChunkedByteBuffer

2016-08-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432284#comment-15432284 ] Guoqiang Li commented on SPARK-17082: - OK > Replace ByteBuffer with ChunkedByteBuffer >

[jira] [Commented] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432279#comment-15432279 ] Guoqiang Li commented on SPARK-17184: - ok > Replace ByteBuf with InputStream >

[jira] [Commented] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-22 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431903#comment-15431903 ] Guoqiang Li commented on SPARK-17184: - OK, I'll post the detailed design document later. > Replace

[jira] [Created] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-22 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-17184: --- Summary: Replace ByteBuf with InputStream Key: SPARK-17184 URL: https://issues.apache.org/jira/browse/SPARK-17184 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-17082) Replace ByteBuffer with ChunkedByteBuffer

2016-08-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-17082: Description: The size of ByteBuffers can not be greater than 2G, should be replaced by

[jira] [Updated] (SPARK-17082) Replace ByteBuffer with ChunkedByteBuffer

2016-08-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-17082: Description: The size of ByteBuffers can not be greater than 2G, should be replaced by Java (was:

[jira] [Updated] (SPARK-17082) Replace ByteBuffer with ChunkedByteBuffer

2016-08-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-17082: Issue Type: Sub-task (was: New Feature) Parent: SPARK-6235 > Replace ByteBuffer with

[jira] [Created] (SPARK-17082) Replace ByteBuffer with ChunkedByteBuffer

2016-08-16 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-17082: --- Summary: Replace ByteBuffer with ChunkedByteBuffer Key: SPARK-17082 URL: https://issues.apache.org/jira/browse/SPARK-17082 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-15 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421048#comment-15421048 ] Guoqiang Li commented on SPARK-6235: Yes, it contains a lot of minor changes, eg: Replace ByteBuffer

[jira] [Comment Edited] (SPARK-6235) Address various 2G limits

2016-08-14 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420554#comment-15420554 ] Guoqiang Li edited comment on SPARK-6235 at 8/15/16 1:53 AM: - [~hvanhovell]

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-14 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420554#comment-15420554 ] Guoqiang Li commented on SPARK-6235: [~hvanhovell] The main changes. 1. Replace DiskStore method

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-11 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418225#comment-15418225 ] Guoqiang Li commented on SPARK-6235: I'm doing this work and I'll put the patch in this month. >

[jira] [Updated] (SPARK-14178) DAGScheduler should get map output statuses directly, not by MapOutputTrackerMaster.getSerializedMapOutputStatuses.

2016-03-26 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-14178: Summary: DAGScheduler should get map output statuses directly, not by

[jira] [Created] (SPARK-14178) Compare Option[String] and String directly

2016-03-26 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-14178: --- Summary: Compare Option[String] and String directly Key: SPARK-14178 URL: https://issues.apache.org/jira/browse/SPARK-14178 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13371) TaskSetManager.dequeueSpeculativeTask compares Option[String] and String directly.

2016-02-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152424#comment-15152424 ] Guoqiang Li commented on SPARK-13371: - [~srowen] Of course not. >

[jira] [Issue Comment Deleted] (SPARK-13371) TaskSetManager.dequeueSpeculativeTask compares Option[String] and String directly.

2016-02-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-13371: Comment: was deleted (was: [~srowen] Of course not.) > TaskSetManager.dequeueSpeculativeTask

[jira] [Commented] (SPARK-13371) TaskSetManager.dequeueSpeculativeTask compares Option[String] and String directly.

2016-02-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152425#comment-15152425 ] Guoqiang Li commented on SPARK-13371: - [~srowen] Of course not. >

[jira] [Updated] (SPARK-13371) TaskSetManager.dequeueSpeculativeTask compares Option[String] and String directly.

2016-02-17 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-13371: Summary: TaskSetManager.dequeueSpeculativeTask compares Option[String] and String directly. (was:

[jira] [Updated] (SPARK-13371) Compare Option[String] and String directly in

2016-02-17 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-13371: Summary: Compare Option[String] and String directly in (was: Compare Option[String] and String

[jira] [Updated] (SPARK-13371) Compare Option[String] and String directly

2016-02-17 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-13371: Description: {noformat} TaskSetManager.dequeueSpeculativeTask compares Option[String] and String

[jira] [Created] (SPARK-13371) Compare Option[String] and String directly

2016-02-17 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-13371: --- Summary: Compare Option[String] and String directly Key: SPARK-13371 URL: https://issues.apache.org/jira/browse/SPARK-13371 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-5575) Artificial neural networks for MLlib deep learning

2015-11-02 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984912#comment-14984912 ] Guoqiang Li commented on SPARK-5575: Hi [~bing], It is under development, and temporarily is private.

[jira] [Comment Edited] (SPARK-5575) Artificial neural networks for MLlib deep learning

2015-11-01 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984809#comment-14984809 ] Guoqiang Li edited comment on SPARK-5575 at 11/2/15 6:59 AM: - Hi All, There is

[jira] [Commented] (SPARK-5575) Artificial neural networks for MLlib deep learning

2015-11-01 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984809#comment-14984809 ] Guoqiang Li commented on SPARK-5575: Hi All, There is a MLP implementation with the Parameter Server

[jira] [Created] (SPARK-10350) Fix SQL Programming Guide

2015-08-29 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-10350: --- Summary: Fix SQL Programming Guide Key: SPARK-10350 URL: https://issues.apache.org/jira/browse/SPARK-10350 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-10350) Fix SQL Programming Guide

2015-08-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-10350: Description:

[jira] [Updated] (SPARK-10350) Fix SQL Programming Guide

2015-08-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-10350: Description:

[jira] [Updated] (SPARK-10350) Fix SQL Programming Guide

2015-08-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-10350: Description:

[jira] [Commented] (SPARK-8276) NPE in YarnClientSchedulerBackend.stop

2015-06-11 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582185#comment-14582185 ] Guoqiang Li commented on SPARK-8276: Can you give more error log of this issue ? NPE

[jira] [Updated] (SPARK-8273) Driver hangs up when yarn shutdown in client mode

2015-06-10 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-8273: --- Affects Version/s: 1.4.0 1.3.1 Driver hangs up when yarn shutdown in client

[jira] [Updated] (SPARK-8273) Driver hangs up when yarn shutdown in client mode

2015-06-09 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-8273: --- Component/s: YARN Driver hangs up when yarn shutdown in client mode

[jira] [Updated] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7934: --- Description: The conf/spark-defaults.conf {noformat} spark.executor.extraJavaOptions -XX:+UseG1GC

[jira] [Updated] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7934: --- Description: The conf/spark-defaults.conf {noformat} spark.executor.extraJavaOptions -XX:+UseG1GC

[jira] [Commented] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564386#comment-14564386 ] Guoqiang Li commented on SPARK-7934: OK, the information is a little less, but I can

[jira] [Updated] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7934: --- Description: The conf/spark-defaults.conf {noformat} spark.executor.extraJavaOptions -XX:+UseG1GC

[jira] [Commented] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564409#comment-14564409 ] Guoqiang Li commented on SPARK-7934: cc [~srowen] In some cases, Spark hangs in

[jira] [Reopened] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li reopened SPARK-7934: In some cases, Spark hangs in yarn-client mode. ---

[jira] [Commented] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-29 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564562#comment-14564562 ] Guoqiang Li commented on SPARK-7934: [~srowen] I want to say that some reasons cause

[jira] [Created] (SPARK-7934) In some cases, Spark hangs in yarn-client mode.

2015-05-28 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-7934: -- Summary: In some cases, Spark hangs in yarn-client mode. Key: SPARK-7934 URL: https://issues.apache.org/jira/browse/SPARK-7934 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-05-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530029#comment-14530029 ] Guoqiang Li commented on SPARK-5556:

[jira] [Reopened] (SPARK-7008) An implementation of Factorization Machine (LibFM)

2015-05-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li reopened SPARK-7008: This jira should not be closed.. An implementation of Factorization Machine (LibFM)

[jira] [Commented] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-04-28 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518602#comment-14518602 ] Guoqiang Li commented on SPARK-5556: I put the latest LDA code in

[jira] [Updated] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-04-28 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-5556: --- Attachment: LDA_test.xlsx Latent Dirichlet Allocation (LDA) using Gibbs sampler

[jira] [Updated] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-04-28 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-5556: --- Attachment: spark-summit.pptx Latent Dirichlet Allocation (LDA) using Gibbs sampler

[jira] [Commented] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-04-28 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518618#comment-14518618 ] Guoqiang Li commented on SPARK-5556: LDA_Gibbs combines the advantages of AliasLDA,

[jira] [Commented] (SPARK-5556) Latent Dirichlet Allocation (LDA) using Gibbs sampler

2015-04-28 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518621#comment-14518621 ] Guoqiang Li commented on SPARK-5556:

[jira] [Updated] (SPARK-7162) spark-shell launcher failure in yarn-client

2015-04-27 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7162: --- Description: {code:none} HADOOP_CONF_DIR=/usr/local/CDH5/hadoop-2.3.0-cdh5.0.1/etc/hadoop/

[jira] [Updated] (SPARK-7162) spark-shell launcher failure in yarn-client

2015-04-27 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7162: --- Description: {code:none} HADOOP_CONF_DIR=/usr/local/CDH5/hadoop-2.3.0-cdh5.0.1/etc/hadoop/

[jira] [Commented] (SPARK-7008) An implementation of Factorization Machine (LibFM)

2015-04-27 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14513856#comment-14513856 ] Guoqiang Li commented on SPARK-7008: [~mengxr] what's your view for [~podongfeng]

[jira] [Updated] (SPARK-7162) Launcher error in yarn-client

2015-04-27 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7162: --- Summary: Launcher error in yarn-client (was: spark-shell launcher error in yarn-client) Launcher

[jira] [Updated] (SPARK-7162) spark-shell launcher failure in yarn-client

2015-04-27 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7162: --- Description: {code:none} HADOOP_CONF_DIR=/usr/local/CDH5/hadoop-2.3.0-cdh5.0.1/etc/hadoop/

[jira] [Created] (SPARK-7162) spark-shell launcher failure in yarn-client

2015-04-27 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-7162: -- Summary: spark-shell launcher failure in yarn-client Key: SPARK-7162 URL: https://issues.apache.org/jira/browse/SPARK-7162 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-7162) spark-shell launcher error in yarn-client

2015-04-27 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7162: --- Summary: spark-shell launcher error in yarn-client (was: spark-shell launcher failure in

[jira] [Commented] (SPARK-7008) An implementation of Factorization Machine (LibFM)

2015-04-24 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512238#comment-14512238 ] Guoqiang Li commented on SPARK-7008: In practice, relative to the {{LBFGS}} ,{{SGD

[jira] [Comment Edited] (SPARK-7008) An implementation of Factorization Machine (LibFM)

2015-04-24 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504435#comment-14504435 ] Guoqiang Li edited comment on SPARK-7008 at 4/25/15 4:16 AM: -

[jira] [Updated] (SPARK-7008) An Implement of Factorization Machine (LibFM)

2015-04-21 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-7008: --- Attachment: FM_convergence_rate.xlsx QQ20150421-1.png QQ20150421-2.png

[jira] [Comment Edited] (SPARK-7008) Implement of Factorization Machine (LibFM)

2015-04-20 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502616#comment-14502616 ] Guoqiang Li edited comment on SPARK-7008 at 4/20/15 10:34 AM: --

[jira] [Commented] (SPARK-6932) A Prototype of Parameter Server

2015-04-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502332#comment-14502332 ] Guoqiang Li commented on SPARK-6932: I have three questions: 1. {code:none} def

[jira] [Commented] (SPARK-3937) Unsafe memory access inside of Snappy library

2015-04-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498189#comment-14498189 ] Guoqiang Li commented on SPARK-3937: [~joshrosen] This bug occurs every time. I'm not

[jira] [Commented] (SPARK-6849) The constructor of GradientDescent should be public

2015-04-12 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491861#comment-14491861 ] Guoqiang Li commented on SPARK-6849: [~srowen] https://github.com/cloudml/zen The

[jira] [Commented] (SPARK-3937) Unsafe memory access inside of Snappy library

2015-04-12 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491857#comment-14491857 ] Guoqiang Li commented on SPARK-3937: Get data: {code:none}wget

[jira] [Commented] (SPARK-6849) The constructor of GradientDescent should be public

2015-04-10 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490642#comment-14490642 ] Guoqiang Li commented on SPARK-6849: [~srowen] This should be a bug. The

[jira] [Commented] (SPARK-3937) Unsafe memory access inside of Snappy library

2015-04-09 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486810#comment-14486810 ] Guoqiang Li commented on SPARK-3937: The bug seems to be caused by

[jira] [Commented] (SPARK-3937) Unsafe memory access inside of Snappy library

2015-04-08 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486763#comment-14486763 ] Guoqiang Li commented on SPARK-3937: I have encountered this issues in spark 1.3 .

[jira] [Updated] (SPARK-5261) In some cases ,The value of word's vector representation is too big

2015-04-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-5261: --- Description: Get data: {code:none} normalize_text() { awk '{print tolower($0);}' | sed -e s/’/'/g

[jira] [Updated] (SPARK-5261) In some cases ,The value of word's vector representation is too big

2015-04-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-5261: --- Description: Get data: {code:none} normalize_text() { awk '{print tolower($0);}' | sed -e s/’/'/g

[jira] [Commented] (SPARK-5261) In some cases ,The value of word's vector representation is too big

2015-04-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481378#comment-14481378 ] Guoqiang Li commented on SPARK-5261: I'm sorry, the after one 's mincount is 100

[jira] [Updated] (SPARK-5261) In some cases ,The value of word's vector representation is too big

2015-04-06 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-5261: --- Description: Get data: {code:none} normalize_text() { awk '{print tolower($0);}' | sed -e s/’/'/g

[jira] [Updated] (SPARK-5261) In some cases ,The value of word's vector representation is too big

2015-04-05 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-5261: --- Description: Get data: {code:none} normalize_text() { awk '{print tolower($0);}' | sed -e s/’/'/g

[jira] [Commented] (SPARK-3625) In some cases, the RDD.checkpoint does not work

2015-04-04 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395613#comment-14395613 ] Guoqiang Li commented on SPARK-3625: Sometimes, when calling the RDD.checkpoint ,

[jira] [Reopened] (SPARK-5261) In some cases ,The value of word's vector representation is too big

2015-04-04 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li reopened SPARK-5261: [~srowen] SPARK-5261 andSPARK-4846 are not the same problem. This is a algorithm error. The

[jira] [Created] (SPARK-6697) PeriodicGraphCheckpointer is not clear Edges.

2015-04-03 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-6697: -- Summary: PeriodicGraphCheckpointer is not clear Edges. Key: SPARK-6697 URL: https://issues.apache.org/jira/browse/SPARK-6697 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3625) In some cases, the RDD.checkpoint does not work

2015-04-03 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395492#comment-14395492 ] Guoqiang Li commented on SPARK-3625: When we run the machine learning and graph

[jira] [Updated] (SPARK-6697) PeriodicGraphCheckpointer is not clear Edges.

2015-04-03 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-6697: --- Attachment: QQ20150403-1.png Web UI Screenshot PeriodicGraphCheckpointer is not clear Edges.

  1   2   3   4   5   >