[
https://issues.apache.org/jira/browse/SPARK-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990006#comment-14990006
]
Glenn Strycker commented on SPARK-3789:
---
I posted a similar question on stackoverflow about a year
Glenn Strycker created SPARK-11387:
--
Summary: minimize shuffles during joins by using existing
partitions and bundling messages
Key: SPARK-11387
URL: https://issues.apache.org/jira/browse/SPARK-11387
[
https://issues.apache.org/jira/browse/SPARK-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979070#comment-14979070
]
Glenn Strycker commented on SPARK-11387:
This ticket may be a particular implementation idea that
[
https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957554#comment-14957554
]
Glenn Strycker commented on SPARK-6235:
---
I don't think so, but I can check. My RDD came from an RDD
[
https://issues.apache.org/jira/browse/SPARK-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker updated SPARK-11004:
---
Description:
Could a feature be added to Spark that would use disk-only MapReduce operations
Glenn Strycker created SPARK-11004:
--
Summary: MapReduce Hive-like join operations for RDDs
Key: SPARK-11004
URL: https://issues.apache.org/jira/browse/SPARK-11004
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948943#comment-14948943
]
Glenn Strycker commented on SPARK-11004:
True, fixing the 2GB will go a long way. However, this
[
https://issues.apache.org/jira/browse/SPARK-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949076#comment-14949076
]
Glenn Strycker commented on SPARK-11004:
Currently we could do the following from withing a linux
[
https://issues.apache.org/jira/browse/SPARK-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949171#comment-14949171
]
Glenn Strycker commented on SPARK-11004:
So maybe we can simplify this idea down to forcing
[
https://issues.apache.org/jira/browse/SPARK-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949187#comment-14949187
]
Glenn Strycker commented on SPARK-11004:
Awesome -- thanks, I'll try that out.
Is there a way to
[
https://issues.apache.org/jira/browse/SPARK-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949076#comment-14949076
]
Glenn Strycker edited comment on SPARK-11004 at 10/8/15 6:12 PM:
-
[
https://issues.apache.org/jira/browse/SPARK-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949076#comment-14949076
]
Glenn Strycker edited comment on SPARK-11004 at 10/8/15 6:13 PM:
-
[
https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906877#comment-14906877
]
Glenn Strycker commented on SPARK-10735:
This appears very similar to a problem I had earlier
[
https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906877#comment-14906877
]
Glenn Strycker edited comment on SPARK-10735 at 9/24/15 7:40 PM:
-
This
[
https://issues.apache.org/jira/browse/SPARK-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904572#comment-14904572
]
Glenn Strycker commented on SPARK-4489:
---
My ticket SPARK-10762 may have just been a user error, but
[
https://issues.apache.org/jira/browse/SPARK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker closed SPARK-10762.
--
This probably isn't completely fixed, but should be a new ticket for casting
ArrayBuffers
[
https://issues.apache.org/jira/browse/SPARK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker resolved SPARK-10762.
Resolution: Not A Problem
Instead of
[
https://issues.apache.org/jira/browse/SPARK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904567#comment-14904567
]
Glenn Strycker commented on SPARK-10762:
Please see the accepted solution to
[
https://issues.apache.org/jira/browse/SPARK-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904571#comment-14904571
]
Glenn Strycker commented on SPARK-1040:
---
My ticket SPARK-10762 may have just been a user error, but
Glenn Strycker created SPARK-10762:
--
Summary: GenericRowWithSchema exception in casting ArrayBuffer to
HashSet in DataFrame to RDD from Hive table
Key: SPARK-10762
URL:
[
https://issues.apache.org/jira/browse/SPARK-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903542#comment-14903542
]
Glenn Strycker commented on SPARK-2737:
---
I am getting a similar error in Spark 1.3.0... see a new
[
https://issues.apache.org/jira/browse/SPARK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903551#comment-14903551
]
Glenn Strycker commented on SPARK-10762:
Is this related?
[
https://issues.apache.org/jira/browse/SPARK-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903541#comment-14903541
]
Glenn Strycker commented on SPARK-1040:
---
I am getting a similar error in Spark 1.3.0... see a new
Glenn Strycker created SPARK-10636:
--
Summary: RDD filter does not work after if..then..else RDD blocks
Key: SPARK-10636
URL: https://issues.apache.org/jira/browse/SPARK-10636
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790681#comment-14790681
]
Glenn Strycker commented on SPARK-10636:
I didn't "forget", I believed that "RDD = if {} else {}
[
https://issues.apache.org/jira/browse/SPARK-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker closed SPARK-10636.
--
> RDD filter does not work after if..then..else RDD blocks
>
[
https://issues.apache.org/jira/browse/SPARK-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741486#comment-14741486
]
Glenn Strycker commented on SPARK-10569:
Playing around with adding additional registrations, I
Glenn Strycker created SPARK-10569:
--
Summary: Kryo serialization fails on sortByKey operation on
registered RDDs
Key: SPARK-10569
URL: https://issues.apache.org/jira/browse/SPARK-10569
Project:
[
https://issues.apache.org/jira/browse/SPARK-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741582#comment-14741582
]
Glenn Strycker commented on SPARK-10569:
Is this issue related to HIVE-7540 or SPARK-2421?
>
[
https://issues.apache.org/jira/browse/SPARK-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741638#comment-14741638
]
Glenn Strycker commented on SPARK-10569:
Note that I am still using 1.3.0. I noticed that
[
https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741639#comment-14741639
]
Glenn Strycker commented on SPARK-10251:
I opened a ticket earlier today that might be related to
[
https://issues.apache.org/jira/browse/SPARK-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741554#comment-14741554
]
Glenn Strycker commented on SPARK-10569:
I'm also seeing the occasional "User class threw
[
https://issues.apache.org/jira/browse/SPARK-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741627#comment-14741627
]
Glenn Strycker commented on SPARK-10569:
It looks very similar to this thread:
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738794#comment-14738794
]
Glenn Strycker commented on SPARK-10493:
Unfortunately we don't have anything past 1.3.0. We're
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736869#comment-14736869
]
Glenn Strycker commented on SPARK-10493:
The RDD I am using has the form ((String, String),
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737001#comment-14737001
]
Glenn Strycker commented on SPARK-10493:
In this example, our RDDs are partitioned with a hash
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737055#comment-14737055
]
Glenn Strycker edited comment on SPARK-10493 at 9/9/15 3:40 PM:
I'm still
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker updated SPARK-10493:
---
Attachment: reduceByKey_example_001.scala
I'm still working on checking unit tests and
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737727#comment-14737727
]
Glenn Strycker commented on SPARK-10493:
I already have that added in my code that I'm testing...
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737735#comment-14737735
]
Glenn Strycker commented on SPARK-10493:
Of course. I have count statements everywhere in order
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737296#comment-14737296
]
Glenn Strycker commented on SPARK-10493:
[~srowen], the code I attached did run correctly.
Glenn Strycker created SPARK-10493:
--
Summary: reduceByKey not returning distinct results
Key: SPARK-10493
URL: https://issues.apache.org/jira/browse/SPARK-10493
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735557#comment-14735557
]
Glenn Strycker commented on SPARK-2620:
---
I am finding similar behavior for a non-case-class RDD...
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735626#comment-14735626
]
Glenn Strycker commented on SPARK-10493:
Thanks for the speedy follow-up, [~frosner]!
I'm
[
https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735653#comment-14735653
]
Glenn Strycker commented on SPARK-10493:
Note: this only seems to be occurring "at scale" so far.
[
https://issues.apache.org/jira/browse/SPARK-8666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker closed SPARK-8666.
-
checkpointing does not take advantage of persisted/cached RDDs
[
https://issues.apache.org/jira/browse/SPARK-8666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker updated SPARK-8666:
--
Description:
I have been noticing that when checkpointing RDDs, all operations are occurring
[
https://issues.apache.org/jira/browse/SPARK-8666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603175#comment-14603175
]
Glenn Strycker commented on SPARK-8666:
---
I added a stackoverflow question to
[
https://issues.apache.org/jira/browse/SPARK-8666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603189#comment-14603189
]
Glenn Strycker commented on SPARK-8666:
---
Looks like this is ticket is a duplicate of
Glenn Strycker created SPARK-8666:
-
Summary: checkpointing does not take advantage of persisted/cached
RDDs
Key: SPARK-8666
URL: https://issues.apache.org/jira/browse/SPARK-8666
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603195#comment-14603195
]
Glenn Strycker commented on SPARK-8582:
---
I didn't see this ticket and made a
[
https://issues.apache.org/jira/browse/SPARK-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker closed SPARK-1885.
-
Resolution: Fixed
User needed to use reduceByKey, not reduce
GraphX reduce function not
[
https://issues.apache.org/jira/browse/SPARK-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker updated SPARK-1883:
--
Summary: spark graph.triplets does not return correct values (was: spark
graphx triplets.map
Glenn Strycker created SPARK-1883:
-
Summary: spark graphx triplets.map does not return correct values
Key: SPARK-1883
URL: https://issues.apache.org/jira/browse/SPARK-1883
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002358#comment-14002358
]
Glenn Strycker commented on SPARK-1883:
---
Sorry, this has been fixed --
[
https://issues.apache.org/jira/browse/SPARK-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Strycker resolved SPARK-1883.
---
Resolution: Fixed
already fixed -- user is running an old version of Spark
spark
Glenn Strycker created SPARK-1885:
-
Summary: GraphX reduce function not working properly -- returns
only 1 element
Key: SPARK-1885
URL: https://issues.apache.org/jira/browse/SPARK-1885
Project: Spark
57 matches
Mail list logo
