Hi, One more update. Sorry When I created a code sample that I shared. I put StringBuilder under the setup function but actually it was on the start bundle function. So far I tested below scenarios - with StringWriter construct object every processElement call - with StringBuilder construct object every processElement call - with StringBuilder construct object every startBundle call (and also tried setLength(0) and delete(0,sb.length() to clean StringBuilder)
None of the cases prevent DF jobs from getting below error. > Shutting down JVM after 8 consecutive periods of measured GC thrashing. > Memory is used/total/max = 4112/5994/5994 MB, GC last/max = 97.36/97.36 %, > #pushbacks=3, gc thrashing=true. Heap dump not written. And also my process rate is 4kps per instance. I would like to hear your suggestions if you have any. Thanks On Wed, Sep 2, 2020 at 6:22 PM Talat Uyarer <tuya...@paloaltonetworks.com> wrote: > I also tried Brian's suggestion to clear stringbuilder by calling delete > with stringbuffer length. No luck. I am still getting the same error > message. Do you have any suggestions ? > > Thanks > > On Wed, Sep 2, 2020 at 3:33 PM Talat Uyarer <tuya...@paloaltonetworks.com> > wrote: > >> If I'm understanding Talat's logic correctly, it's not necessary to reuse >>> the string builder at all in this case. >> >> Yes. I tried it too. But DF job has the same issue. >> >> >> On Wed, Sep 2, 2020 at 3:17 PM Kyle Weaver <kcwea...@google.com> wrote: >> >>> > It looks like `writer.setLength(0)` may actually allocate a new >>> buffer, and then the buffer may also need to be resized as the String >>> grows, so you could be creating a lot of orphaned buffers very quickly. I'm >>> not that familiar with StringBuilder, is there a way to reset it and re-use >>> the existing capacity? Maybe `writer.delete(0, writer.length())` [1]? >>> >>> If I'm understanding Talat's logic correctly, it's not necessary to >>> reuse the string builder at all in this case. >>> >>> On Wed, Sep 2, 2020 at 3:11 PM Brian Hulette <bhule...@google.com> >>> wrote: >>> >>>> That error isn't exactly an OOM, it indicates the JVM is spending a >>>> significant amount of time in garbage collection. >>>> >>>> It looks like `writer.setLength(0)` may actually allocate a new buffer, >>>> and then the buffer may also need to be resized as the String grows, so you >>>> could be creating a lot of orphaned buffers very quickly. I'm not that >>>> familiar with StringBuilder, is there a way to reset it and re-use the >>>> existing capacity? Maybe `writer.delete(0, writer.length())` [1]? >>>> >>>> [1] >>>> https://stackoverflow.com/questions/242438/is-it-better-to-reuse-a-stringbuilder-in-a-loop >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_242438_is-2Dit-2Dbetter-2Dto-2Dreuse-2Da-2Dstringbuilder-2Din-2Da-2Dloop&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=BkW1L6EF7ergAVYDXCo-3Vwkpy6qjsWAz7_GD7pAR8g&m=8xskmxTZ2EbxwBknWfeIiV2kEsXsu9dzjWT_yG6A0s4&s=ZL6S353ZUzPRmxrPo8Sei_mdxsWDxs4Km2RwwiwefEU&e=> >>>> >>>> On Wed, Sep 2, 2020 at 3:02 PM Talat Uyarer < >>>> tuya...@paloaltonetworks.com> wrote: >>>> >>>>> Sorry for the wrong import. You can see on the code I am using >>>>> StringBuilder. >>>>> >>>>> On Wed, Sep 2, 2020 at 2:55 PM Ning Kang <ni...@google.com> wrote: >>>>> >>>>>> Here is a question answered on StackOverflow: >>>>>> https://stackoverflow.com/questions/27221292/when-should-i-use-javas-stringwriter >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_27221292_when-2Dshould-2Di-2Duse-2Djavas-2Dstringwriter&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=BkW1L6EF7ergAVYDXCo-3Vwkpy6qjsWAz7_GD7pAR8g&m=mVBqxC5kNOARPduF-c17S1VnIw8gwS6alvgONJKfheY&s=ggveahdPKo3vaAhADvjz4ucjndSmzyOZ8FPBvJ_0oZQ&e=> >>>>>> >>>>>> Could you try using StringBuilder instead since the usage is not >>>>>> appropriate for a StringWriter? >>>>>> >>>>>> >>>>>> On Wed, Sep 2, 2020 at 2:49 PM Talat Uyarer < >>>>>> tuya...@paloaltonetworks.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have an issue with String Concatenating. You can see my code >>>>>>> below.[1] I have a step on my df job which is concatenating strings. But >>>>>>> somehow when I use that step my job starts getting jvm restart errors. >>>>>>> >>>>>>> Shutting down JVM after 8 consecutive periods of measured GC >>>>>>>> thrashing. Memory is used/total/max = 4112/5994/5994 MB, GC last/max = >>>>>>>> 97.36/97.36 %, #pushbacks=3, gc thrashing=true. Heap dump not written. >>>>>>> >>>>>>> >>>>>>> And also I try to use Avro rather than String. When I use Avro, it >>>>>>> works fine without any issue. Do you have any suggestions? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> [1] https://dpaste.com/7RTV86WQC >>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__dpaste.com_7RTV86WQC&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=BkW1L6EF7ergAVYDXCo-3Vwkpy6qjsWAz7_GD7pAR8g&m=mVBqxC5kNOARPduF-c17S1VnIw8gwS6alvgONJKfheY&s=eSd0NcP8fw5BOZlSXtUMRfYuGWlN-gcXENVwgCmrapY&e=> >>>>>>> >>>>>>> >>>>>>>
