yeah it looks like this upgrade will require a little fix.
Basically, now they need implicits to WritableFactory instead of implicit
to a Writable.
It will be code compatible as long as people name stuff explicitly (like
RDD[(Int,Int)]). but if it s a generic, i dont think it will be code
compatible despite what they say (which is our case).
btw one of shortcomings that was in technical debt was the limited scope of
K keys we can actually successfully read or save (it is now pretty much
limited by 4 types or so we try to name explicitly there). Maybe this is an
apt time to redesign this a bit as well. 'cause spark api has the same
problems.
On Mon, Mar 23, 2015 at 11:06 AM, Dmitriy Lyubimov <dlie...@gmail.com>
wrote:
> lemme read this issue really quick.
>
> This looks like a redundant double-contract. Why require implicit
> conversions if they are already requiring explicit types? And vice versa.
>
>
> On Sun, Mar 22, 2015 at 10:17 AM, Pat Ferrel <p...@occamsmachete.com>
> wrote:
>
>> Due to a bug in spark we have a nasty work around for Spark 1.2.1 so I’m
>> trying 1.3.0.
>>
>> Hoever they have redesigned the rdd.saveAsSequenceFile in
>> SequenceFileRDDFunctions. The class now expects K and V Writables to be
>> supplied in the constructor:
>>
>> class SequenceFileRDDFunctions[K <% Writable: ClassTag, V <% Writable :
>> ClassTag](
>> self: RDD[(K, V)],
>> _keyWritableClass: Class[_ <: Writable], // <=========new
>> _valueWritableClass: Class[_ <: Writable]) // <========new
>> extends Logging
>> with Serializable {
>>
>> as explained in the commit log:
>>
>> [SPARK-4795][Core] Redesign the "primitive type => Writable" implicit
>> APIs to make them be activated automatically Try to redesign the "primitive
>> type => Writable" implicit APIs to make them be activated automatically and
>> without breaking binary compatibility. However, this PR will breaking the
>> source compatibility if people use `xxxToXxxWritable` occasionally. See the
>> unit test in `graphx`. Author: zsxwing Closes #3642 from zsxwing/SPARK-4795
>> and squashes the following commits:
>>
>> Since Andy, Gokhan, and Dmitriy have been messing with the Key type
>> recently I didn’t want to plow ahead with this before consulting. It
>> appears that the Writable classes need to be available to the constructor
>> when the RDD is written. This breaks all instances of
>> rdd.saveAsSequenceFile in Mahout.
>>
>> Where is the best place to fix this?
>
>
>
