FairScheduler + single resource policy

2015-08-17 Thread William Slacum
Hi all, I'm using Hadoop 2.6.0 after upgrading from Hadoop 2.2.0. Previously, I didn't mess with any vcore settings as we weren't doing anything special with containers and it seemed happy enough to only consider memory as a contended resource. This is consistent with the behavior described in the

Re: Custom comparator when using Kryo serializer for MapReduce serialization

2015-08-19 Thread William Slacum
In a general sense, if you can guarantee that your objects serialize in lexicographical order, then you should be able to do a comparator on the raw bytes themselves without any interpretation. On Wed, Aug 19, 2015 at 5:21 AM, Yaron Gonen wrote: > Hi all, > (I'm using Hadoop 1.2.1) > I'm using K

Re: Custom comparator when using Kryo serializer for MapReduce serialization

2015-08-21 Thread William Slacum
. > How can I guarantee that in a distributed system? > > On Wed, Aug 19, 2015 at 8:06 PM, William Slacum wrote: > >> In a general sense, if you can guarantee that your objects serialize in >> lexicographical order, then you should be able to do a comparator on th