Re: OutOfMemoryError in mirror maker

2015-06-28 Thread tao xiao
That is so cool. Thank you On Sun, 28 Jun 2015 at 04:29 Guozhang Wang wrote: > Tao, I have added you to the contributor list of Kafka so you can assign > tickets to yourself now. > > I will review the patch soon. > > Guozhang > > On Thu, Jun 25, 2015 at 2:54 AM, tao xiao wrote: > > > Patch upda

Re: OutOfMemoryError in mirror maker

2015-06-27 Thread Guozhang Wang
Tao, I have added you to the contributor list of Kafka so you can assign tickets to yourself now. I will review the patch soon. Guozhang On Thu, Jun 25, 2015 at 2:54 AM, tao xiao wrote: > Patch updated. please review > > On Mon, 22 Jun 2015 at 12:24 tao xiao wrote: > > > Yes, you are right. W

Re: OutOfMemoryError in mirror maker

2015-06-25 Thread tao xiao
Patch updated. please review On Mon, 22 Jun 2015 at 12:24 tao xiao wrote: > Yes, you are right. Will update the patch > On Mon, Jun 22, 2015 at 12:16 PM Jiangjie Qin > wrote: > >> Should we still store the value bytes when logAsString is set to TRUE and >> only store the length when logAsString

Re: OutOfMemoryError in mirror maker

2015-06-21 Thread tao xiao
Yes, you are right. Will update the patch On Mon, Jun 22, 2015 at 12:16 PM Jiangjie Qin wrote: > Should we still store the value bytes when logAsString is set to TRUE and > only store the length when logAsString is set to FALSE. > > On 6/21/15, 7:29 PM, "tao xiao" wrote: > > >The patch I submitt

Re: OutOfMemoryError in mirror maker

2015-06-21 Thread Jiangjie Qin
Should we still store the value bytes when logAsString is set to TRUE and only store the length when logAsString is set to FALSE. On 6/21/15, 7:29 PM, "tao xiao" wrote: >The patch I submitted did the what you suggested. It store the size only >and print it out when error occurs. > >On Mon, Jun 2

Re: OutOfMemoryError in mirror maker

2015-06-21 Thread tao xiao
The patch I submitted did the what you suggested. It store the size only and print it out when error occurs. On Mon, Jun 22, 2015 at 5:26 AM Jiangjie Qin wrote: > Yes, we can expose a user callback in MM, just like we did for rebalance > listener. > I still think ErrorLoggingCallback needs some

Re: OutOfMemoryError in mirror maker

2015-06-21 Thread Jiangjie Qin
Yes, we can expose a user callback in MM, just like we did for rebalance listener. I still think ErrorLoggingCallback needs some change, though. Can we only store the value bytes when logAsString is set to true? That looks more reasonable to me. Jiangjie (Becket) Qin On 6/21/15, 3:02 AM, "tao xia

Re: OutOfMemoryError in mirror maker

2015-06-21 Thread tao xiao
Yes, I agree with that. It is even better if we can supply our own callback. For people who want to view the content of message when failure they still can do so On Sun, Jun 21, 2015 at 2:20 PM Guozhang Wang wrote: > Hi Tao / Jiangjie, > > I think a better fix here may be not letting MirrorMaker

Re: OutOfMemoryError in mirror maker

2015-06-20 Thread Guozhang Wang
Hi Tao / Jiangjie, I think a better fix here may be not letting MirrorMakerProducerCallback to extend from ErrorLoggingCallback, but rather change the ErrorLoggingCallback itself as it defeats the usage of logAsString, which I think is useful for a general error logging purposes. Rather we can let

Re: OutOfMemoryError in mirror maker

2015-06-17 Thread tao xiao
Thank you for the reply. Patch submitted https://issues.apache.org/jira/browse/KAFKA-2281 On Mon, 15 Jun 2015 at 02:16 Jiangjie Qin wrote: > Hi Tao, > > Yes, the issue that ErrorLoggingCallback keeps value as local variable is > known for a while and we probably should fix it as the value is no

Re: OutOfMemoryError in mirror maker

2015-06-14 Thread Jiangjie Qin
Hi Tao, Yes, the issue that ErrorLoggingCallback keeps value as local variable is known for a while and we probably should fix it as the value is not used except logging the its size. Can you open a ticket and maybe also submit a patch? For unreachable objects I don¹t think it is memory leak. As

OutOfMemoryError in mirror maker

2015-06-13 Thread tao xiao
Hi, I am using mirror maker in trunk to replica data across two data centers. While the destination broker was having busy load and unresponsive the send rate of mirror maker was very low and the available producer buffer was quickly filled up. At the end mirror maker threw OOME. Detailed exceptio