Re: kafka.utils.DoublyLinkedList

2015-02-04 Thread Jiangjie Qin
Hi Jay, The approach you suggested is exactly what we did at beginning. But Jun mentioned a case in RB that could cause problem. "Suppose there are 2 target partitions. Offset 1,2,4,5 can be routed to partition 1 and offset 3 can be routed to partition 2. When all messages are sent, you will see

Re: kafka.utils.DoublyLinkedList

2015-02-04 Thread Jay Kreps
Yeah I think the Java LL is worse than that, you actually need to point at it with an iterator and call remove(). I didn't realize the scala ll was deprecated. Okay, makes sense. Would be good to move that test out of UtilsTest.scala, though. But actually, stepping back, can you guys document/expl

Re: kafka.utils.DoublyLinkedList

2015-02-04 Thread Joel Koshy
Jay, My understanding is that java linked lists (even though they are backed by a doubly linked list) support O(1) deletion only if you have the index of the object. If you do a remove(object) it becomes O(n). As for scala lists - the scaladocs seem to indicate that the doubly linked lists are dep

Re: kafka.utils.DoublyLinkedList

2015-02-04 Thread Jay Kreps
Hey Jiangjie, Both the scala and java doubly linked lists allow O(1) deletion--it isn't obvious because the api is slightly different from the CS101 linked list but they do. Are you sure those wouldn't work? Can we remove this class and use those? Also the convention for unit tests is if we have

Re: kafka.utils.DoublyLinkedList

2015-02-03 Thread Jiangjie Qin
We added this because in redesigned mirror maker we need a raw linked list so we can removed an acked offset from a linked list in O(1). Java and Scala linked list won¹t work in that case. It was initially put in side mirror maker as a private class, but later on we also want to have unit test for

kafka.utils.DoublyLinkedList

2015-02-03 Thread Jay Kreps
How come we added this? Is it better than the linked list in java and scala? Also it was added with only one test which is actually put inside UtilsTest.scala which is meant to house tests for Utils.scala. -Jay