Re: [Spark Core] unhashable type: 'dict' during shuffle step

2017-07-18 Thread Josh Holbrook
not keeping my hopes up... Thanks, --Josh On Tue, Jul 18, 2017 at 3:17 PM, Josh Holbrook <josh.holbr...@fusion.net> wrote: > Hello! > > I'm running into a very strange issue with pretty much no hits on the > internet, and I'm hoping someone here can give me some protips! At

[Spark Core] unhashable type: 'dict' during shuffle step

2017-07-18 Thread Josh Holbrook
Hello! I'm running into a very strange issue with pretty much no hits on the internet, and I'm hoping someone here can give me some protips! At this point, I'm at a loss. This is a little long-winded, but hopefully you'll indulge me. Background: I'm currently trying to port some existing spark

Re: Running Spark und YARN on AWS EMR

2017-07-17 Thread Josh Holbrook
I just ran into this issue! Small world. As far as I can tell, by default spark on EMR is completely untuned, but it comes with a flag that you can set to tell EMR to autotune spark. In your configuration.json file, you can add something like: { "Classification": "spark", "Properties":