Thanks Matthias again for all the suggestions .. On Monday 31 October 2016, Matthias J. Sax <matth...@confluent.io> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Just one comment to add: > > > it produces messages based > >> on 0.9.0 which obviously cannot be consumed by 0.10.0 brokers. > > Strictly speaking, this is not correct. Brokers are compatible to > older versions of clients (not the other way round though). For 0.9 > client writing to 0.10 brokers, it only happens that the client does > not set the timestamp field, but this is not an issue by itself. If > you consume a topic like this, this will also work. > > The issue you observed is a special combination of different things. A > situation as describe about, plus Streams property. In Streams, input > record timestamps determine output record timestamps and the used 0.10 > producer requires a valid TS value. Thus, a Streams application fails > for this case. > > Btw: if you still want to use 0.9 producer client, you could also > change broker setting from "create time" to "append time". Using > append time, regardless of the producer behavior, a valid TS will be > set for all (newly) written records. Keep in mind, that this no global > but a per-topic broker setting. > > - -Matthias > > > On 10/30/16 1:15 AM, Debasish Ghosh wrote: > > I think I found out what happened .. I was installing Kafka under > > DC/OS on AWS and following this doc > > https://dcos.io/docs/1.8/usage/tutorials/kafka/ for kafka > > installation. This works fine and installs kafka version 0.10.0. > > > > But in the example where it shows how to produce and consume > > messages, this doc says the following .. > > > > core@ip-10-0-6-153 ~ $ docker run -it mesosphere/kafka-client > >> root@7d0aed75e582:/bin# echo "Hello, World." | > >> ./kafka-console-producer.sh --broker-list KAFKA_ADDRESS:PORT > >> --topic topic1 > > > > > > The problem is the docker run of kafka-client pulls in version > > 0.9.0. When I use kafka-console-producer from this client, it > > produces messages based on 0.9.0 which obviously cannot be consumed > > by 0.10.0 brokers. > > > > Thanks for pointing in this direction .. I think it will be fixed > > if I install a 0.10.0 client in its place. > > > > regards. > > > > On Sunday 30 October 2016, Matthias J. Sax <matth...@confluent.io > <javascript:;>> > > wrote: > > > > The simplest way should be to check the java classpath. > > > > Insert an > > > > echo $CLASSPATH > > > > at the end of bin/kafka-run-class.sh > > > > Than run bin/kafka-console-producer.sh with no argument. > > > > You should see the classpath be printed out. Look for > > 'kafka-clients-XXX.jar' -- XXX will be the version number. > > > > > > -Matthias > > > > > > On 10/29/16 12:11 AM, Debasish Ghosh wrote: > >>>> Hello Mathias - > >>>> > >>>> Regarding .. > >>>> > >>>> In case you do have 0.10 brokers, it might however happen, > >>>> that bin/kafka-console-producer.sh > >>>>> does use 0.9 producer. > >>>> > >>>> > >>>> How can I check this ? > >>>> > >>>> Thanks! > >>>> > >>>> On Sat, Oct 29, 2016 at 12:23 PM, Debasish Ghosh > >>>> <ghosh.debas...@gmail.com <javascript:;> <javascript:;>> wrote: > >>>> > >>>>> I agree .. the problem is DC/OS still ships the older > >>>>> version. Let me check if I can upgrade this .. > >>>>> > >>>>> Thanks! > >>>>> > >>>>> On Sat, Oct 29, 2016 at 12:21 PM, Matthias J. Sax > >>>>> <matth...@confluent.io <javascript:;> <javascript:;>> wrote: > >>>>> > >>>> Btw: I would highly recommend to use Kafka 0.10.1 -- there > >>>> are many new Streams feature and usability improvements and > >>>> bug fixes. > >>>> > >>>> -Matthias > >>>> > >>>> On 10/28/16 11:42 PM, Matthias J. Sax wrote: > >>>>>>>> That sounds reasonable. However, I am wondering how > >>>>>>>> your Streams application can connect to 0.9 broker in > >>>>>>>> the first place. Streams internally uses standard > >>>>>>>> Kafka clients, and those are not backward compatible. > >>>>>>>> Thus, the 0.10 Streams clients should not be able to > >>>>>>>> connect to 0.9 broker. > >>>>>>>> > >>>>>>>> In case you do have 0.10 brokers, it might however > >>>>>>>> happen, that bin/kafka-console-producer.sh does use > >>>>>>>> 0.9 producer. Broker are backward compatible, thus, a > >>>>>>>> 0.9 producer can write to 0.10 broker (and in this > >>>>>>>> case record TS would be invalid). While I assume that > >>>>>>>> in you local environment you are using 0.10 > >>>>>>>> bin/kafka-console-produer.sh and thus all works > >>>>>>>> fine. > >>>>>>>> > >>>>>>>> > >>>>>>>> -Matthias > >>>>>>>> > >>>>>>>> > >>>>>>>> On 10/28/16 11:00 PM, Debasish Ghosh wrote: > >>>>>>>>> Hello Mathias - > >>>>>>>> > >>>>>>>>> Thanks a lot for the response. I think what may be > >>>>>>>>> happening is a version mismatch between the > >>>>>>>>> development & deployment versions of Kafka. The > >>>>>>>>> Kafka streams application that I developed uses > >>>>>>>>> 0.10.0 based libraries. And my local environment > >>>>>>>>> contains a server installation of the same version. > >>>>>>>>> Hence it works ok in my local environment. > >>>>>>>> > >>>>>>>>> But the DC/OS 1.8 deploys 0.9.0 of Kafka when I > >>>>>>>>> install the service through DC/OS cli. And I use > >>>>>>>>> this version to load records into the input topic. > >>>>>>>>> And try to consume using the deployed streams > >>>>>>>>> application which I developed using 0.10.0. Hence > >>>>>>>>> the producer did not put the timestamp while the > >>>>>>>>> consumer expects to have one. > >>>>>>>> > >>>>>>>>> I need to check if 0.10.x is available for DC/OS > >>>>>>>>> .. > >>>>>>>> > >>>>>>>>> Thanks again for your suggestions. > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Sat, Oct 29, 2016 at 3:23 AM, Matthias J. Sax > >>>>>>>>> <matth...@confluent.io <javascript:;> <javascript:;>> wrote: > >>>>>>>> > >>>>>>>>> Hey, > >>>>>>>> > >>>>>>>>> we just added a new FAQ entry for upcoming CP 3.2 > >>>>>>>>> release that answers your question. I just c&p it > >>>>>>>>> here. More concrete answer below. > >>>>>>>> > >>>>>>>>>>>> If you get an exception similar to the one > >>>>>>>>>>>> shown below, there are multiple possible > >>>>>>>>>>>> causes: > >>>>>>>>>>>> > >>>>>>>>>>>> Exception in thread "StreamThread-1" > >>>>>>>>>>>> java.lang.IllegalArgumentException: Invalid > >>>>>>>>>>>> timestamp -1 at > >>>>>>>>>>>> > >>>>>>>>>>>> org.apache.kafka.clients.producer.ProducerRecord.<init>(Pro > duc > > > >>>>>>>>>>>> > erRe > >>>> > >>>>>>>>>>>> > > c > >>>>>>>> > >>>>>>>>>>>> > >>>> ord > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> .java:60) > >>>>>>>>>>>> > >>>>>>>>>>>> This error means that the timestamp extractor > >>>>>>>>>>>> of your Kafka Streams application failed to > >>>>>>>>>>>> extract a valid timestamp from a record. > >>>>>>>>>>>> Typically, this points to a problem with the > >>>>>>>>>>>> record (e.g., the record does not contain a > >>>>>>>>>>>> timestamp at all), but it could also indicate > >>>>>>>>>>>> a problem or bug in the timestamp extractor > >>>>>>>>>>>> used by the application. > >>>>>>>>>>>> > >>>>>>>>>>>> When does a record not contain a valid > >>>>>>>>>>>> timestamp: > >>>>>>>>>>>> > >>>>>>>>>>>> If you are using the default > >>>>>>>>>>>> ConsumerRecordTimestampExtractor, it is most > >>>>>>>>>>>> likely that your records do not carry an > >>>>>>>>>>>> embedded timestamp (embedded record > >>>>>>>>>>>> timestamps got introduced in Kafka's message > >>>>>>>>>>>> format in Kafka 0.10). This might happen, if > >>>>>>>>>>>> you consume a topic that is written by old > >>>>>>>>>>>> Kafka producer clients (ie, version 0.9 or > >>>>>>>>>>>> earlier) or third party producer clients. A > >>>>>>>>>>>> common situation where this may happen is > >>>>>>>>>>>> after upgrading your Kafka cluster from 0.9 > >>>>>>>>>>>> to 0.10, where all the data that was > >>>>>>>>>>>> generated with 0.9 is not compatible with the > >>>>>>>>>>>> 0.10 message format. If you are using a > >>>>>>>>>>>> custom timestamp extractor, make sure that > >>>>>>>>>>>> your extractor is robust to missing > >>>>>>>>>>>> timestamps in your records. For example, you > >>>>>>>>>>>> can return a default or estimated timestamp > >>>>>>>>>>>> if you cannot extract a valid timestamp > >>>>>>>>>>>> (maybe the timstamp field in your data is > >>>>>>>>>>>> just missing). You can also switch to > >>>>>>>>>>>> processing time semantics via > >>>>>>>>>>>> WallclockTimestampExtractor; whether such a > >>>>>>>>>>>> fallback is an appropriate response to this > >>>>>>>>>>>> situation depends on your use case. However, > >>>>>>>>>>>> as a first step you should identify and fix > >>>>>>>>>>>> the root cause for why such problematic > >>>>>>>>>>>> records were written to Kafka in the first > >>>>>>>>>>>> place. In a second step you may consider > >>>>>>>>>>>> applying workarounds (as described above) > >>>>>>>>>>>> when dealing with such records (for example, > >>>>>>>>>>>> if you need to process those records after > >>>>>>>>>>>> all). Another option is to regenerate the > >>>>>>>>>>>> records with correct timestamps and write > >>>>>>>>>>>> them to a new Kafka topic. > >>>>>>>>>>>> > >>>>>>>>>>>> When the timestamp extractor causes the > >>>>>>>>>>>> problem: > >>>>>>>>>>>> > >>>>>>>>>>>> In this situation you should debug and fix > >>>>>>>>>>>> the erroneous extractor. If the extractor is > >>>>>>>>>>>> built into Kafka, please report the bug to > >>>>>>>>>>>> the Kafka developer mailing list at > >>>>>>>>>>>> d...@kafka.apache.org <javascript:;> <javascript:;> (see > >>>>>>>>>>>> instructions > >>>>>>>>>>>> http://kafka.apache.org/contact); in the > >>>>>>>>>>>> meantime, you may write a custom timestamp > >>>>>>>>>>>> extractor that fixes the problem and > >>>>>>>>>>>> configure your application to use that > >>>>>>>>>>>> extractor for the time being. > >>>>>>>> > >>>>>>>> > >>>>>>>>> To address you questions more concretely: > >>>>>>>> > >>>>>>>>> 1. Yes an no: Yes, for any new data you write to > >>>>>>>>> you topic. No, for any already written data that > >>>>>>>>> does not have a valid timestamp set 2. Default is > >>>>>>>>> creating time 3. Config parameter > >>>>>>>>> "message.timestamp.type") It's a broker side per > >>>>>>>>> topic setting (however, be aware that Java > >>>>>>>>> KafkaProducer does verify the timestamp locally > >>>>>>>>> before sending the message to the broker, thus on > >>>>>>>>> -1 there will be the client side exception you did > >>>>>>>>> observe( 4. I assume that you do consumer different > >>>>>>>>> topic with different TS fields in you records. > >>>>>>>> > >>>>>>>>> Also have a look at: > >>>>>>>>> http://docs.confluent.io/current/streams/concepts.html#time > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>> > > > >>>>>>>>> > - -Matthias > >>>>>>>> > >>>>>>>> > >>>>>>>>> On 10/28/16 5:42 AM, Debasish Ghosh wrote: > >>>>>>>>>>>> I am actually using 0.10.0 and NOT 0.10.1 as > >>>>>>>>>>>> I mentioned in the last mail. And I am using > >>>>>>>>>>>> Kafka within a DC/OS cluster under AWS. > >>>>>>>>>>>> > >>>>>>>>>>>> The version that I mentioned works ok is on > >>>>>>>>>>>> my local machine using a local Kafka > >>>>>>>>>>>> installation. And it works for both single > >>>>>>>>>>>> broker and multi broker scenario. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks. > >>>>>>>>>>>> > >>>>>>>>>>>> On Fri, Oct 28, 2016 at 1:37 PM, Debasish > >>>>>>>>>>>> Ghosh <ghosh.debas...@gmail.com <javascript:;> > >>>>>>>>>>>> <javascript:;>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hello - > >>>>>>>>>>>>> > >>>>>>>>>>>>> I am a beginner in Kafka .. with my first > >>>>>>>>>>>>> Kafka streams application .. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I have a streams application that reads > >>>>>>>>>>>>> from a topic, does some transformation on > >>>>>>>>>>>>> the data and writes to another topic. The > >>>>>>>>>>>>> record that I manipulate is a CSV record. > >>>>>>>>>>>>> > >>>>>>>>>>>>> It runs fine when I run it on a local > >>>>>>>>>>>>> Kafka instance. > >>>>>>>>>>>>> > >>>>>>>>>>>>> However when I run it on an AWS cluster, I > >>>>>>>>>>>>> get the following exception when I try to > >>>>>>>>>>>>> produce the transformed record into the > >>>>>>>>>>>>> target topic. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Exception in thread "StreamThread-1" > >>>>>>>>>>>>> java.lang.IllegalArgumentException: > >>>>>>>>>>>>> Invalid timestamp -1 at > >>>>>>>>>>>>> org.apache.kafka.clients.producer.ProducerRecord.<init> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > (ProducerRecord.java:60) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals.SinkNode. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > process(SinkNode.java:72) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > StreamTask.forward(StreamTask.java:338) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > ProcessorContextImpl.forward(ProcessorContextImpl.java:187) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > at > >>>>>>>>>>>>> org.apache.kafka.streams.kstream.internals.KStreamMapValue > s$ > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > KStreamMapProcessor.process(KStreamMapValues.java:42) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals.ProcessorNode > .pr > > > >>>>>>>>>>>>> > oces > >>>> > >>>>>>>>>>>>> > > s > >>>>>>>> > >>>>>>>>>>>>> > >>>> ( > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> ProcessorNode.java:68) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > StreamTask.forward(StreamTask.java:338) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > ProcessorContextImpl.forward(ProcessorContextImpl.java:187) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > at > >>>>>>>>>>>>> org.apache.kafka.streams.kstream.internals.KStreamPassThro > ugh > > > >>>>>>>>>>>>> > $ > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > KStreamPassThroughProcessor.process(KStreamPassThrough.java:34) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals.ProcessorNode > .pr > > > >>>>>>>>>>>>> > oces > >>>> > >>>>>>>>>>>>> > > s > >>>>>>>> > >>>>>>>>>>>>> > >>>> ( > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> ProcessorNode.java:68) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > StreamTask.forward(StreamTask.java:351) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > ProcessorContextImpl.forward(ProcessorContextImpl.java:192) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > at > >>>>>>>>>>>>> org.apache.kafka.streams.kstream.internals.KStreamBranch$ > >>>>>>>>>>>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > KStreamBranchProcessor.process(KStreamBranch.java:46) at > >>>>>>>>>>>>> > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals.ProcessorNode > .pr > > > >>>>>>>>>>>>> > oces > >>>> > >>>>>>>>>>>>> > > s > >>>>>>>> > >>>>>>>>>>>>> > >>>> ( > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> ProcessorNode.java:68) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > StreamTask.forward(StreamTask.java:338) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > ProcessorContextImpl.forward(ProcessorContextImpl.java:187) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > at > >>>>>>>>>>>>> org.apache.kafka.streams.kstream.internals.KStreamMapValue > s$ > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > KStreamMapProcessor.process(KStreamMapValues.java:42) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals.ProcessorNode > .pr > > > >>>>>>>>>>>>> > oces > >>>> > >>>>>>>>>>>>> > > s > >>>>>>>> > >>>>>>>>>>>>> > >>>> ( > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> ProcessorNode.java:68) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > StreamTask.forward(StreamTask.java:338) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > ProcessorContextImpl.forward(ProcessorContextImpl.java:187) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>> > > at org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> SourceNode.process(SourceNode.java:64) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > StreamTask.process(StreamTask.java:174) at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals.StreamThread. > run > > > >>>>>>>>>>>>> > Loop > >>>> > >>>>>>>>>>>>> > > ( > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> StreamThread.java:320) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > StreamThread.run(StreamThread.java:218) > >>>>>>>>>>>>> > >>>>>>>>>>>>> Looks like the timestamp passed to the > >>>>>>>>>>>>> ProducerRecord is -1, though I am not > >>>>>>>>>>>>> passing any timestamp explicitly. I am not > >>>>>>>>>>>>> sure why this happens. But I see that the > >>>>>>>>>>>>> Javadoc for ProducerRecord says the > >>>>>>>>>>>>> following .. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The record also has an associated > >>>>>>>>>>>>> timestamp. If the user did not provide a > >>>>>>>>>>>>>> timestamp, the producer will stamp the > >>>>>>>>>>>>>> record with its current time. The > >>>>>>>>>>>>>> timestamp eventually used by Kafka > >>>>>>>>>>>>>> depends on the timestamp type configured > >>>>>>>>>>>>>> for the topic. If the topic is configured > >>>>>>>>>>>>>> to use CreateTime, the timestamp in the > >>>>>>>>>>>>>> producer record will be used by the > >>>>>>>>>>>>>> broker. If the topic is configured to > >>>>>>>>>>>>>> use LogAppendTime, the timestamp in the > >>>>>>>>>>>>>> producer record will be overwritten by > >>>>>>>>>>>>>> the broker with the broker local time > >>>>>>>>>>>>>> when it appends the message to its log. > >>>>>>>>>>>>>> In either of the cases above, the > >>>>>>>>>>>>>> timestamp that has actually been used > >>>>>>>>>>>>>> will be returned to user in > >>>>>>>>>>>>>> RecordMetadata > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1. Will this problem be solved if I > >>>>>>>>>>>>> configure the topic with LogAppendTime or > >>>>>>>>>>>>> CreateTime explicitly ? 2. What is the > >>>>>>>>>>>>> default setting of this property in a newly > >>>>>>>>>>>>> created topic ? 3. How do I change it (what > >>>>>>>>>>>>> is the name of the property to be set) ? > >>>>>>>>>>>>> 4. Any idea why I face this problem in the > >>>>>>>>>>>>> cluster mode but not in the local mode ? > >>>>>>>>>>>>> > >>>>>>>>>>>>> BTW I am using 0.10.1. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Any help / pointer will be appreciated ? > >>>>>>>>>>>>> > >>>>>>>>>>>>> regards. > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- Debasish Ghosh > >>>>>>>>>>>>> http://manning.com/ghosh2 > >>>>>>>>>>>>> http://manning.com/ghosh > >>>>>>>>>>>>> > >>>>>>>>>>>>> Twttr: @debasishg Blog: > >>>>>>>>>>>>> http://debasishg.blogspot.com Code: > >>>>>>>>>>>>> http://github.com/debasishg > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- Debasish Ghosh http://manning.com/ghosh2 > >>>>> http://manning.com/ghosh > >>>>> > >>>>> Twttr: @debasishg Blog: http://debasishg.blogspot.com > >>>>> Code: http://github.com/debasishg > >>>>> > >>>> > >>>> > >>>> > >> > > > > > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - https://gpgtools.org > > iQIcBAEBCgAGBQJYFs69AAoJECnhiMLycopPLmkQAKGTE0GRxduQ7VHXCrBC2EJR > ZiObYfPaI1dqkPz0m/rEgPAcUpBB855uzs9Oa77hngfT3xfwa05kDDrH+eEgsYgI > YdqLt02inPd11UKztj+i8x1mQAqVB4IGzCQExmGgX9yDWpXpVgv+t+3gByVaoryK > 18LxmbVHOeIiUEpsjyXeknlZUomOZ1fO30nSLCL+zmOjLjOnHicd5IYTOrRuBIGR > J2bTvI0bxuDPJlIDKW7/nHC9OfInHcyO2B8k0LPK7IFgDlb/z+ICazbiIwjzkwrP > fsuLla11N8+FkH3GNZOtP/gAgVJXmhVerZtAKn85xOGajQcs5o3MWJV3YCBKXtOq > k+gqxP4GbWUIP/Zz+5QxD6bRugj0I86XiaMOrBfUjkXBtnDJCdWhDAP3WIZMYoFu > ZbT33kakDMPRLOlxEiPkM2bPYe99SqzSh+v/Z2fiI8sqsGVgkPcwEkAWZupjy6b4 > /c2CHXK1wnhoRpOKpQUoKparZFp5v6cY56/vPjxU32soXyG0rx/0Jd8JHv7x2AFD > 0xi7PvA6QvIF9m8RdMddrMjwQ5VW1yeYL20ZDf7mGQr+S96rHg7zqdYhwjkgWERx > lBJQ8Q6sHF7XeRZvoShCxZ06BtD7LTokintAUjaV3+jX/pvxNGgBrQ7aD7Lyw4k/ > tnDi1J+ISJanK15EGPg6 > =gy51 > -----END PGP SIGNATURE----- > -- Sent from my iPhone
