Hi, We are proposing to build connectors for memcache first and then use it for Couchbase. The connector for memcache will be build as a IOTransform and then it can be used for other memcache implementations including Couchbase.
1. As Source input will be a key(String / byte[]), output will be a KV<key, value> where key - String / byte[] value - String / byte[] Spymemcached supports a multi-get operation where it takes a bunch of keys and retrieves the associated values, the input PCollection<key> can be bundled into multiple batches and each batch can be submitted via the multi-get operation. PCollection<KV<byte[], byte[]>> values = MemCacheIO .withConfig() .read() .withKey(PCollection<byte[]>); 2. As Sink input will be a KV<key, value>, output will be none or probably a boolean indicating the outcome of the operation //write MemCacheIO .withConfig() .write() .withEntries(PCollection<KV<byte[],byte[]>>); Implementation plan 1. Develop Memcache connector with 'set' and 'add' operation 2. Then develop other operations 3. Use Memcache connector for Couchbase Thanks @Ismael for help Please, let us know your views. Madhu Borkar
