[
https://issues.apache.org/jira/browse/DL-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767616#comment-15767616
]
ASF GitHub Bot commented on DL-99:
----------------------------------
GitHub user sijie opened a pull request:
https://github.com/apache/incubator-distributedlog/pull/82
DL-99 - DL-111
This is a umbrella merge for DL-99 to DL-111. Individual commit in this
pull request is associated with one DL jira.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sijie/incubator-distributedlog
squashed_merge/DL-111
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-distributedlog/pull/82.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #82
----
commit 9ff22f4239754d25fb3c7238cf25cf453f0ebbd6
Author: Dave Rusek <[email protected]>
Date: 2016-07-18T19:23:49Z
DL-99: Enable thriftmux for distributedlog smoketest reader
commit 785ff1cdfb515882f934d4bd79fe34bb39cd95a9
Author: Sijie Guo <[email protected]>
Date: 2016-07-29T04:10:33Z
DL-101: Improve session expire handling on fetching log segments for readers
This change focuses on improving the session expire handling on fetching
log segments for readers.
The log segment management in DL is now done by 3 parts.
- a LogSegmentMetadataStore (one per namespace instance): it is used for
fetching the log segments from log segment metadata store (ZooKeeper). it
doesn't do any caching.
- a LogSegmentMetadataCache (one per namespace instance): it is a guava
cache based metadata cache. it maintains a mapping between log segment metadata
path and the log segment metadata. it manages the cache for the log segments
that will be accessed in this namespace instance. it doesn't manage the
sequence of the log segments for streams.
- a PerStreamLogSegmentCache for each BKLogHandler. the log segment cache
is per stream. it maintains the sequence of the log segments.
BKLogWriteHandler doesn't watch the log segment changes. It fetches minimal
number of log segments when it is created and fetches the full list of log
segments for truncations. New log segments will be added to the per stream log
segment cache with log segment rolling.
BKLogReadHandler watch the log segments changes and only notify when the
list of log segments is changed. the session handling which is specific to the
metadata store is hidden to the implementations of LogSegmentMetadataStore.
The change tries to cleanup bunch of unused methods in
BKLog{Read,Write}Handler too.
commit 10ff44510437d4087df5dc4775d968e174df83b0
Author: Leigh Stewart <[email protected]>
Date: 2016-07-29T04:36:49Z
DL-102: Add routing service to write proxy server side
this change is to add getOwner rpc in write proxy. so we can change the
client side to get owner from write proxy first for routing service. in this
way, we can start experiementing any resource placement algorithms.
commit 9f80812ab4293de2f7073642f7ef452b6a88f8f6
Author: Sijie Guo <[email protected]>
Date: 2016-07-29T05:02:09Z
DL-103: DL client can use server-side routing service to retrieve
ownerships for streams
If the dl client configures a routing service finagle name, the client will
use this to retrieve ownerships for streams.
commit 75774256dbaa5fcccb2e0d319e5d7d272a976901
Author: Jordan Bull <[email protected]>
Date: 2016-08-09T18:36:53Z
DL-104: BKClientReadTimeout can be equal to getReadLACLongPollTimeout so
current config validates
commit afe64639b871b4dbc564f904fc4f522c0de334d3
Author: Yiming Zang <[email protected]>
Date: 2016-08-10T23:18:53Z
DL-105: Make compression stats available per stream
commit 36a07f6b1d5d3fde395687f97ee10fb9376b5d7b
Author: Yiming Zang <[email protected]>
Date: 2016-08-12T17:30:39Z
DL-106: Use namespace after it is closed will throw AlreadyClosedException
commit d156c1d6223034fceba90750febc42d25e389d5c
Author: Phillip Su <[email protected]>
Date: 2016-08-23T00:47:28Z
DL-107: Added unregistering gauges for distributedlog-core and
distributedlog-benchmark
commit e18cd48c2f9a9d03dfc7f46e96e74ebaa1471659
Author: Yiming Zang <[email protected]>
Date: 2016-09-17T00:35:51Z
DL-108: Log rate limiting more clearly
commit a743b44f4614154c67cc4d6a98a9003d80062d70
Author: Sijie Guo <[email protected]>
Date: 2016-10-27T17:18:52Z
DL-109: Add a tool to find the stream containing the given ledger
commit 5f50a41b3883e2107befa95f774687deae9d62fe
Author: Sijie Guo <[email protected]>
Date: 2016-11-22T00:17:51Z
DL-110: Write control record if necessary when roll a log segment
commit 0d2e20c7c9189dec7909ae67f4a62443a9aa40bc
Author: Sijie Guo <[email protected]>
Date: 2016-11-22T01:30:13Z
DL-111: ReadAhead Cache should cache entries rather than records
Current readahead cache cache records. So it will be a lot of callbacks
(function calls) when polling a record off the readahead cache. Most of the cpu
cycles are unnecessarily spent
on function calls on polling records off the readahead cache. It is the
throughput bottleneck for a DL reader.
This change is to change ReadAhead cache to cache entries rather than
records. Defer the deserilization of records later on when the reader wants to
access the records. It also make
the cache more efficient to reduce the memory footprint.
commit 63b62709e982a8cdabfa0b15ae88c956e71606d0
Author: Sijie Guo <[email protected]>
Date: 2016-11-22T01:35:40Z
DL-112: Improve SyncReader to access readahead cache directly
This changes the implementation of SyncReader to let it access the read
ahead cache directly, rather
than calling AsyncReader to fetch the entries. It would reduce the function
calls that SyncReader spends
on reading individual records.
commit 27157ec59d30e3e519e5bdef9dfb157c5185ffea
Author: Sijie Guo <[email protected]>
Date: 2016-11-22T01:41:26Z
DL-113: Improve the sync reader benchmark
Add more logging information for calcuating the stats for catching up reads
----
> Enable thrift mux on DL smoketest reader
> ----------------------------------------
>
> Key: DL-99
> URL: https://issues.apache.org/jira/browse/DL-99
> Project: DistributedLog
> Issue Type: Bug
> Components: distributedlog-benchmark
> Reporter: Sijie Guo
> Fix For: 0.4.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
