Hi Matei, I've clarified the documentation to include this information in this pull request. Can you take a look?
https://github.com/apache/incubator-spark/pull/206 On Mon, Nov 25, 2013 at 5:03 PM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > Yeah, if you just say spark.serializer to Kryo, it will use it for all > these things. > > Matei > > On Nov 25, 2013, at 4:59 PM, Andrew Ash <and...@andrewash.com> wrote: > > How do you know Spark doesn't also use Kryo for shuffled files? Are there > metrics or logs somewhere that make you believe it's normal Java > serialization? > > > On Mon, Nov 25, 2013 at 4:46 PM, Mayuresh Kunjir < > mayuresh.kun...@gmail.com> wrote: > >> This shows how to serialize user classes. I wanted Spark to serialize all >> shuffle files and object files using Kryo. How can I specify that? Or would >> that be done by default if I just set spark.serializer to kryo? >> >> >> >> >> On Mon, Nov 25, 2013 at 7:42 PM, Matei Zaharia >> <matei.zaha...@gmail.com>wrote: >> >>> Did you look through >>> http://spark.incubator.apache.org/docs/latest/tuning.html#data-serialization?It >>> shows an example of how to register classes with Kryo. In particular, in >>> your Registrator, you can use kryo.register(yourClass, new YourSerializer) >>> to pass a custom serializer too. >>> >>> Matei >>> >>> On Nov 25, 2013, at 4:25 PM, Mayuresh Kunjir <mayuresh.kun...@gmail.com> >>> wrote: >>> >>> Hi Spark users, >>> >>> This has probably been answered before, but I could not locate it. I >>> understand from the tuning guide that using Kryo serialization for shuffles >>> improves the performance. I would like to know how to register the Kryo >>> serializer. Apart from the shuffles, my standalone application needs to >>> store and retrieve a few object files as well. I would really appreciate >>> any pointers on registering Kryo serializer for both these serialization >>> tasks. >>> >>> Thanks and regards, >>> ~Mayuresh >>> >>> >>> >>> >> > >
