GitHub user sijie opened a pull request:
https://github.com/apache/incubator-distributedlog/pull/92
DL-115 ~ DL-164
This is cherry-pick merge twitter branch's changes for DL-115 ~ DL-164.
Individual commit is associated with a DL jira.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sijie/incubator-distributedlog
sijie/squash_merge_twitter_changes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-distributedlog/pull/92.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #92
----
commit 00be3e58d9a4a4e48989c8e45825a0c4911f9cdc
Author: Yiming Zang <yz...@twitter.com>
Date: 2016-11-16T22:14:52Z
DL-115: fix force get log segment logic
Summary:
We should remove the ledger closed check:
- force to get log segments if the reader has been idle for a while after
open the ledger
- idleReaderWarnThreshold should be 2x larger than readLACLongPollTimeout
commit 9d467a60a8c2ad2c0acc73afbfb66c232baa0b46
Author: Yiming Zang <yz...@twitter.com>
Date: 2016-11-30T22:16:14Z
DL-116: Add tool for deleting subscriber from subscription store
Test Plan:
1. manually create znode for subscribers resume point
2. use the tool to delete the subscriberId
commit b91d49a89e7c9133eb2807ced2e7dbb8aba72e02
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-01T01:14:05Z
DL-117: Stream metadata store
This change is to abstract the zookeeper operations into a stream metadata
store, so we can replace zookeeper with other metadata store easily.
So the metadata operations in distributedlog now are managed by 3 classes:
- LogMetadataStore : it is the namespace metadata store : it manages the
location (uri) mapping for streams and handle namespace operations.
- LogStreamMetadataStore: it is the stream metadata store : it manages the
metadata for a single stream, such as managing read/write lock,
retriving/creating stream metadata, deleting metadata and such.
- LogSegmentMetadataStore: it is the segment metadata store : it manages
the log segment metadata for individual log segment.
LogMetadataStore and LogSegmentMetadataStore are already there. This change
focus on LogStreamMetadataStore
Changed:
* abstract all the zookeeper metadata operation in log handlers to
LogStreamMetadataStore
* remove disabling max tx id santify check, as maxTxId update is part of
the metadata update transaction
Not changed:
the name of ZKLogMetadataForReader and ZKLogMetadataForWriter are not
changed. I will send out a change to rename these two classes as they are not
related to zookeeper anymore.
commit dc4548bebcd8c8ce680fb14cfaf203dd6605fa29
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-01T02:17:46Z
DL-118: Stream metadata store followup - rename ZKLogMetadata* to
LogMetadata*
As the followup change, this change is to remove 'ZK' from 'ZKLogMetadata*'
class
commit 34fa16b1d7ab1ae887b1114e8e2aabdffe16608a
Author: Yiming Zang <yz...@twitter.com>
Date: 2016-12-01T02:21:08Z
DL-119: Fix the logging on closing readahead worker
commit 0591d067f05617bc534a662b6f9a014192cbe3a5
Author: Jordan Bull <jb...@twitter.com>
Date: 2016-12-13T19:11:03Z
DL-157: resource placement for write proxy
commit 98a29a5807fd7d42d2bb765be84527d4b2fb5dbc
Author: Yiming Zang <yz...@twitter.com>
Date: 2016-12-28T22:49:50Z
DL-158: update truncation status for all completed log segments
update turncation status for all completed log segments, when purge
truncated log segments we will leave one complete log segment.
commit 27c04f37b1b564a06d9105acb1849a6d98449ca9
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-28T22:51:54Z
DL-195: ReadAhead Improvement (part 1) - Interface for
LogSegmentEntryReader and LogSegmentEntryWriter
Create interface for log segment entry reader and writer to abstract data
write/read operations.
commit 7a977972200da9e86f4557bb002ce16cb60d236a
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-28T23:09:38Z
DL-159: ReadAhead Improvement (part 2) - New ReadAhead Reader using the
LogSegmentEntryReader interface
Provide a new ReadAhead reader using the log segment entry reader
interface. It does reading entries in a batch in parallel for batches, rather
than reading entries in batch by batch. This would help mitigate the slow
bookie problem.
The core change is the new ReadAheadEntryReader.
commit c7751804ef728c39d54b19e52f2db48d7fba9f65
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-29T00:27:38Z
DL-160: Remove 'DLSN' suffix from async and sync readers
commit 47622a6e45a34492ec50fcae79fb052dba06c6a4
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-29T00:48:47Z
DL-161: Rename ledger id to log segment id
Along with providing log segment store related interfaces, clean up the
naming from 'ledger id' to 'log segment id'. So it would become clear when we
can plugin different log segment store.
commit d871e6570fe49cab56fcec37d3a8a6bd91afbe4d
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-29T01:09:57Z
DL-162: Use log segment entry store interface
- Use log segment entry store interface
- Delete the old readahead implementation
commit cfc049cd045b73ca96fb328877b0acf4e75c95ab
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-29T01:47:12Z
DL-163: clean up direct zookeeper and bookkeeper usage and use
metadata/data store abstraction
- introduce NamespaceDriver class to manage metadata/data store abstractions
- clean up direct zookeeper and bookkeeper usage and use metadata/data
store abstraction
- separate MetadataAccessor from DistributedLogManager
- remove deprecated classes
commit 63d6bde1929085780171e63cc5b0c95581daa564
Author: Sijie Guo <sij...@twitter.com>
Date: 2016-12-29T03:09:59Z
DL-164: Create stream operation should not be submitted by StreamImpl
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
