Hi AL, kafka deals in blobs, so you generally have to manage serialization / deserialization at the producer + consumer level. kafka-python's SimpleProducer and SimpleConsumer classes are fairly naive and operate exclusively on bytes, so if you use those you will have to serialize before producing, and deserialize after consuming. `dump` and `loads` from json or msgpack or equivalent should work fine.
Note that the new KafkaConsumer class exposes the `deserializer_class` configuration option, which operates the same as the official java client library. Unfortunately, there is no such class for the producer side in kafka-python (though an unfinished WIP exists at https://github.com/mumrah/kafka-python/pull/333). So `KafkaConsumer(..., deserializer_class=json.loads)` would be sufficient to consume directly into python objects if you've produced as json. HTH -Dana (kafka-python maintainer) On Thu, Aug 13, 2015 at 3:14 PM, Sa Li <sal...@gmail.com> wrote: > Hi, All > > I have a question about kafka-python producer, here is the record I have > > id (uuid) | sensor_id (character) | timestamp | period (int) | current > (numeric) | date_received | factor (bigint) > "75da661c-bd5c-40e3-8691-9034f34262e3” | “ff0057” | "2013-03-21 > 11:44:00-07” | 60 | 0.1200 | "2013-03-26 14:40:51.829-07” | > 7485985 > > I am getting data from database and publishing to kafka, I am having the > error of timestamp & decimal serialization, can’t just publish each record > as a list. I am thinking to convert each record to son object, before I do > that, anyone knows more straightforward way to publish directly to a list > (kafka consumer can read each record as a list or dictionary? > > thanks > > AL
