[
https://issues.apache.org/jira/browse/SPARK-21849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21849:
Issue Type: Improvement (was: Bug)
> Make the serializer function more robust
>
DjvuLee created SPARK-21849:
---
Summary: Make the serializer function more robust
Key: SPARK-21849
URL: https://issues.apache.org/jira/browse/SPARK-21849
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120959#comment-16120959
]
DjvuLee commented on SPARK-21682:
-
Yes, our company also faced with this scalability problem, the driver
[
https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106292#comment-16106292
]
DjvuLee commented on SPARK-21547:
-
Ok, I will try and posted the result later.
> Spark cleaner cost
[
https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103459#comment-16103459
]
DjvuLee commented on SPARK-21547:
-
Yes, I agree that this has a relationship with the work, but doing
[
https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21547:
Description:
Spark Streaming sometime cost so many time deal with cleaning, and this can
become worse
[
https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21547:
Description:
Spark Streaming sometime cost so many time deal with cleaning, and this can
become worse
[
https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103005#comment-16103005
]
DjvuLee commented on SPARK-21547:
-
17/07/27 11:29:51 INFO TaskSetManager: Finished task 169.0 in stage
[
https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21547:
Description: Spark Streaming sometime cost so many time deal with cleaning,
and this can become worse when
DjvuLee created SPARK-21547:
---
Summary: Spark cleaner cost too many time
Key: SPARK-21547
URL: https://issues.apache.org/jira/browse/SPARK-21547
Project: Spark
Issue Type: Bug
Components:
[
https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21383:
Summary: YARN can allocate too many executors (was: YARN can allocate to
many executors)
> YARN can
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Affects Version/s: (was: 2.2.1)
2.3.0
> Consider Executor's memory usage when
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049965#comment-16049965
]
DjvuLee commented on SPARK-21082:
-
Data locality, input size for task, scheduling order affect a lot,
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049933#comment-16049933
]
DjvuLee edited comment on SPARK-21082 at 6/15/17 2:47 AM:
--
Not a really fast
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049933#comment-16049933
]
DjvuLee commented on SPARK-21082:
-
Not a really fast node and slow node problem.
Even all the nodes have
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Description:
Spark Scheduler do not consider the memory usage during dispatch tasks, this
can lead to
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048835#comment-16048835
]
DjvuLee commented on SPARK-21082:
-
Yes, one of the reason why Spark do not balance tasks well enough is
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048653#comment-16048653
]
DjvuLee edited comment on SPARK-21082 at 6/14/17 3:15 AM:
--
[~srowen] This
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048654#comment-16048654
]
DjvuLee commented on SPARK-21082:
-
My idea is try to consider the BlockManger information when scheduling
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048653#comment-16048653
]
DjvuLee commented on SPARK-21082:
-
[~srowen] This situation occurred when the partition number is larger
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Description:
Spark Scheduler do not consider the memory usage during dispatch tasks, this
can lead to
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048146#comment-16048146
]
DjvuLee commented on SPARK-21082:
-
If this feature is a good suggestion(we encounter this problem in
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Description:
Spark Scheduler do not consider the memory usage during dispatch tasks, this
can lead to
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Description: Spark Scheduler do not consider the memory usage during
dispatch tasks, this can lead to
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Description: When we cache the
> Consider Executor's memory usage when scheduling task
>
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Component/s: Scheduler
> Consider Executor's memory usage when scheduling task
>
DjvuLee created SPARK-21082:
---
Summary: Consider the Executor's Memory usage when scheduling task
Key: SPARK-21082
URL: https://issues.apache.org/jira/browse/SPARK-21082
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
Summary: Consider Executor's memory usage when scheduling task (was:
Consider the Executor's Memory
[
https://issues.apache.org/jira/browse/SPARK-21064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046475#comment-16046475
]
DjvuLee commented on SPARK-21064:
-
The defalut value for `spark.port.maxRetries` is 100, but we use the
DjvuLee created SPARK-21064:
---
Summary: Fix the default value bug in
NettyBlockTransferServiceSuite
Key: SPARK-21064
URL: https://issues.apache.org/jira/browse/SPARK-21064
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040633#comment-16040633
]
DjvuLee commented on SPARK-18085:
-
[~vanzin] the procedure of loading the history summary page is still a
[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16038210#comment-16038210
]
DjvuLee commented on SPARK-18085:
-
[~vanzin] I want to try your branch. Does all the information is the
[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-18085:
Yes, you're right. I just want to impact as few as possible.
> Better History Server scalability for many
[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896652#comment-15896652
]
DjvuLee commented on SPARK-18085:
-
"A separate jar file" means we generate a new jar file for the history
[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896107#comment-15896107
]
DjvuLee commented on SPARK-19823:
-
[~zsxwing] Can you have a look at?
> Support Gang Distribution of
[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896096#comment-15896096
]
DjvuLee edited comment on SPARK-19823 at 3/5/17 7:19 AM:
-
When Spark distributes
[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896096#comment-15896096
]
DjvuLee edited comment on SPARK-19823 at 3/5/17 7:10 AM:
-
When Spark distributes
[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896096#comment-15896096
]
DjvuLee edited comment on SPARK-19823 at 3/5/17 7:10 AM:
-
When Spark distributes
[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896096#comment-15896096
]
DjvuLee edited comment on SPARK-19823 at 3/5/17 7:10 AM:
-
When Spark distributes
[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896096#comment-15896096
]
DjvuLee commented on SPARK-19823:
-
When Spark distributes tasks to Executors, it uses a Round-Robin way,
[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896097#comment-15896097
]
DjvuLee commented on SPARK-19823:
-
If this is a good advice, I will give a Pull Request.
> Support Gang
DjvuLee created SPARK-19823:
---
Summary: Support Gang Distribution of Task
Key: SPARK-19823
URL: https://issues.apache.org/jira/browse/SPARK-19823
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896070#comment-15896070
]
DjvuLee commented on SPARK-18085:
-
[~vanzin] Thanks for your reply!
Does your new solution will generate
[
https://issues.apache.org/jira/browse/SPARK-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896067#comment-15896067
]
DjvuLee commented on SPARK-19821:
-
Currently, when the disk is just read-only, we will just throw out the
[
https://issues.apache.org/jira/browse/SPARK-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-19821:
Description:
java.io.FileNotFoundException:
DjvuLee created SPARK-19821:
---
Summary: Throw out the Read-only disk information when create file
for Shuffle
Key: SPARK-19821
URL: https://issues.apache.org/jira/browse/SPARK-19821
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894074#comment-15894074
]
DjvuLee commented on SPARK-18085:
-
[~vanzin] This is a nice design.
There is not much information about
[
https://issues.apache.org/jira/browse/SPARK-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853630#comment-15853630
]
DjvuLee commented on SPARK-17300:
-
[~rdblue] Is there any fix for this issue later?
>
DjvuLee created SPARK-19327:
---
Summary: Improve the partition method when read data from jdbc
Key: SPARK-19327
URL: https://issues.apache.org/jira/browse/SPARK-19327
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-19239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-19239:
Summary: Check the lowerBound and upperBound whether equal None in jdbc API
(was: Check the lowerBound
[
https://issues.apache.org/jira/browse/SPARK-19239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-19239:
Description: When we use the ``jdbc`` in pyspark, if we check the
lowerBound and upperBound, we can give a
DjvuLee created SPARK-19239:
---
Summary: Check the lowerBound and upperBound equal None in jdbc API
Key: SPARK-19239
URL: https://issues.apache.org/jira/browse/SPARK-19239
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731403#comment-15731403
]
DjvuLee edited comment on SPARK-18778 at 12/8/16 7:46 AM:
--
When I just run the
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731455#comment-15731455
]
DjvuLee commented on SPARK-18778:
-
[~srowen] [~andrewor14] can you have a look at?
> Fix the Scala
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731431#comment-15731431
]
DjvuLee edited comment on SPARK-18778 at 12/8/16 7:38 AM:
--
I give a fix in the
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731431#comment-15731431
]
DjvuLee edited comment on SPARK-18778 at 12/8/16 7:38 AM:
--
I give a fix in the
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731431#comment-15731431
]
DjvuLee edited comment on SPARK-18778 at 12/8/16 7:35 AM:
--
I give a fix in the
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731431#comment-15731431
]
DjvuLee commented on SPARK-18778:
-
I give a fix in the https://github.com/apache/spark/pull/16210
> Fix
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731403#comment-15731403
]
DjvuLee commented on SPARK-18778:
-
When I just run the ./bin/spark-shell under our environment, the
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731403#comment-15731403
]
DjvuLee edited comment on SPARK-18778 at 12/8/16 7:25 AM:
--
When I just run the
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-18778:
Affects Version/s: 1.6.1
2.0.2
> Fix the Scala classpath in the spark-shell
>
[
https://issues.apache.org/jira/browse/SPARK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-18778:
Description:
Failed to initialize compiler: object scala.runtime in compiler mirror not
found.
** Note
DjvuLee created SPARK-18778:
---
Summary: Fix the Scala classpath in the spark-shell
Key: SPARK-18778
URL: https://issues.apache.org/jira/browse/SPARK-18778
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-18181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685568#comment-15685568
]
DjvuLee commented on SPARK-18181:
-
[~barrybecker4] can you reproduce this on the spark2.x version？
>
[
https://issues.apache.org/jira/browse/SPARK-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685451#comment-15685451
]
DjvuLee commented on SPARK-18528:
-
I just test your example, but it works.
>>>
[
https://issues.apache.org/jira/browse/SPARK-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee closed SPARK-17500.
---
Resolution: Not A Bug
> The DiskBytesSpilled metric in ExternalMerger && ExternalGroupBy is not right
>
[
https://issues.apache.org/jira/browse/SPARK-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-17500:
Description: The DiskBytesSpilled metric in ExternalMerger &&
ExternalGroupBy is increased by file size in
[
https://issues.apache.org/jira/browse/SPARK-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-17500:
Summary: The DiskBytesSpilled metric in ExternalMerger && ExternalGroupBy
is not right (was: The
DjvuLee created SPARK-17500:
---
Summary: The DiskBytesSpilled metric in ExternalMerger &&
ExternalGroupBy is wrong
Key: SPARK-17500
URL: https://issues.apache.org/jira/browse/SPARK-17500
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-3630:
---
Comment: was deleted
(was: How much data do you test? we encounter this error in our production.
Our data
[
https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430215#comment-15430215
]
DjvuLee edited comment on SPARK-3630 at 8/22/16 7:10 AM:
-
Can I know how much data
[
https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430215#comment-15430215
]
DjvuLee commented on SPARK-3630:
How much data do you test? we encounter this error in our production.
[
https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430217#comment-15430217
]
DjvuLee commented on SPARK-3630:
How much data do you test? we encounter this error in our production.
[
https://issues.apache.org/jira/browse/SPARK-11054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952225#comment-14952225
]
DjvuLee commented on SPARK-11054:
-
Since the default value for spark.akka.heartbeat.interval or
DjvuLee created SPARK-11054:
---
Summary: WARN ReliableDeliverySupervisor: Association with remote
system
Key: SPARK-11054
URL: https://issues.apache.org/jira/browse/SPARK-11054
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-11054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952653#comment-14952653
]
DjvuLee commented on SPARK-11054:
-
I am very sorry for set the wrong Priority!
I knew this problem
[
https://issues.apache.org/jira/browse/SPARK-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14877397#comment-14877397
]
DjvuLee commented on SPARK-10717:
-
change to
final class NioBlockTransferService(conf: SparkConf,
DjvuLee created SPARK-10717:
---
Summary: remove the with Loging in the NioBlockTransferService
Key: SPARK-10717
URL: https://issues.apache.org/jira/browse/SPARK-10717
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661207#comment-14661207
]
DjvuLee commented on SPARK-4105:
Is there any progress on this bug?
I have the same
[
https://issues.apache.org/jira/browse/SPARK-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316393#comment-14316393
]
DjvuLee commented on SPARK-5739:
Yes, 1M maybe enough for the Kmeans algorithm.
But if we
[
https://issues.apache.org/jira/browse/SPARK-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316430#comment-14316430
]
DjvuLee commented on SPARK-5739:
Ok, Got it, I will look the code for more detail.
I
[
https://issues.apache.org/jira/browse/SPARK-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317384#comment-14317384
]
DjvuLee commented on SPARK-5739:
Yes, I do not explain cleanly. What I mean is that we can
[
https://issues.apache.org/jira/browse/SPARK-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-5739:
---
Summary: Size exceeds Integer.MAX_VALUE in File Map (was: Size exceeds
Integer.MAX_VALUE in FileMap)
Size
[
https://issues.apache.org/jira/browse/SPARK-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315845#comment-14315845
]
DjvuLee commented on SPARK-5739:
the data is generated by the example KMeansDataGenerator
DjvuLee created SPARK-5739:
--
Summary: Size exceeds Integer.MAX_VALUE in FileMap
Key: SPARK-5739
URL: https://issues.apache.org/jira/browse/SPARK-5739
Project: Spark
Issue Type: Bug
Affects
[
https://issues.apache.org/jira/browse/SPARK-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-5739:
---
Description:
I just run the kmeans algorithm using a random generate data,but occurred this
problem after
[
https://issues.apache.org/jira/browse/SPARK-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-5375:
---
Description:
In the ConnectionManager.scala file, there is three thread pool:
handleMessageExecutor,
DjvuLee created SPARK-5375:
--
Summary: Specify more clearly about the max thread meaning in the
ConnectionManager
Key: SPARK-5375
URL: https://issues.apache.org/jira/browse/SPARK-5375
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-5375:
---
Description:
In the ConnectionManager.scala file, there is three thread pool:
handleMessageExecutor,
[
https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073244#comment-14073244
]
DjvuLee commented on SPARK-1112:
Does anyone test in version0.9.2，I found it also failed ,
[
https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064965#comment-14064965
]
DjvuLee commented on SPARK-2156:
I see this fixed in the spark branch-0.9 in the github,
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061804#comment-14061804
]
DjvuLee commented on SPARK-2138:
[~piotrszul] In my opinion, if your task size if bigger
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061843#comment-14061843
]
DjvuLee commented on SPARK-2138:
Oh, So can we improve this better?
The KMeans
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062026#comment-14062026
]
DjvuLee commented on SPARK-2138:
In my experiment, I set the akka.frameSize=200, my data
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062163#comment-14062163
]
DjvuLee commented on SPARK-2138:
oh, I am a little sorry that I write some mistaken in my
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048488#comment-14048488
]
DjvuLee commented on SPARK-2138:
@[~piotrszul] Thanks for your test, I will also test this
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046597#comment-14046597
]
DjvuLee commented on SPARK-2138:
If this bug fixed, shall we closed this issue? [~mengxr]
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036892#comment-14036892
]
DjvuLee commented on SPARK-2138:
Thanks very much! I am very glad to see that I reported
DjvuLee created SPARK-2138:
--
Summary: The KMeans algorithm in the MLlib can lead to the
Serialized Task size become bigger and bigger
Key: SPARK-2138
URL: https://issues.apache.org/jira/browse/SPARK-2138
[
https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-2138:
---
Description:
When the algorithm running at certain stage, when running the reduceBykey()
algorithm, It can
1 - 100 of 101 matches
Mail list logo
