[jira] [Created] (ACCUMULO-4077) upgrade to thrift 0.9.3
Eric Newton created ACCUMULO-4077: - Summary: upgrade to thrift 0.9.3 Key: ACCUMULO-4077 URL: https://issues.apache.org/jira/browse/ACCUMULO-4077 Project: Accumulo Issue Type: Improvement Components: client, master, proxy, tserver Reporter: Eric Newton Priority: Minor Fix For: 1.8.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (ACCUMULO-4077) upgrade to thrift 0.9.3
[ https://issues.apache.org/jira/browse/ACCUMULO-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Newton reassigned ACCUMULO-4077: - Assignee: Eric Newton > upgrade to thrift 0.9.3 > --- > > Key: ACCUMULO-4077 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4077 > Project: Accumulo > Issue Type: Improvement > Components: client, master, proxy, tserver >Reporter: Eric Newton >Assignee: Eric Newton >Priority: Minor > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (ACCUMULO-4076) Remove generated javadocs from docs/
[ https://issues.apache.org/jira/browse/ACCUMULO-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4076. - Resolution: Fixed > Remove generated javadocs from docs/ > > > Key: ACCUMULO-4076 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4076 > Project: Accumulo > Issue Type: Sub-task > Components: build >Reporter: Josh Elser >Assignee: Christopher Tubbs > Fix For: 1.6.5, 1.7.1, 1.8.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > In running some preliminary Yetus-driven tests, it was found (with the help > of [~aw]!) that we still dump the generate javadocs into docs/ instead of > something like docs/target. This caused Yetus to get a bit confused when > computing the effective diff of some patch being tested as suddenly there > were tons of changes from the original repository. > We stopped bundling the javadocs in our tarballs a while back (which, in > talking with [~ctubbsii] on IRC) is probably why they went to this > non-standard location. > We should make sure our generated javadocs are being put in a sane location, > likely the default location for the maven plugin itself is sufficient and > update the docs with any amended steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-3509) Scanner lock cause Tablet lock, hence preventing idle scans from being swept, hence blocking SimpleTimer thread
[ https://issues.apache.org/jira/browse/ACCUMULO-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060694#comment-15060694 ] Christopher Tubbs commented on ACCUMULO-3509: - Ping for update. Do you have the PR available? > Scanner lock cause Tablet lock, hence preventing idle scans from being swept, > hence blocking SimpleTimer thread > > > Key: ACCUMULO-3509 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3509 > Project: Accumulo > Issue Type: Bug > Components: tserver >Affects Versions: 1.6.0 >Reporter: marco polo >Assignee: marco polo > Fix For: 1.8.0 > > > Synchronization with Tablet$Scanner via a read() will block close() being > called via the sweep method in TabletServer. As a result, the SimpleTimer > thread does not continue, and idle threads grow until the scan completes. > My patch, which is forthcoming, converts synchronized methods to use a fair > lock. If the lock is held by a read call, the close call will attempt to > obtain it, time out, and return indicating a close was not successful. The > sweep will continue, and the SimpleTimer thread will respawn later, > attempting closure on those Tablets at a later time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-3835) TabletServerBatchReaderIterator concurrency contention on early close()
[ https://issues.apache.org/jira/browse/ACCUMULO-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060688#comment-15060688 ] Josh Elser commented on ACCUMULO-3835: -- +1 on a backport at first glance. > TabletServerBatchReaderIterator concurrency contention on early close() > --- > > Key: ACCUMULO-3835 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3835 > Project: Accumulo > Issue Type: Bug > Components: client >Affects Versions: 1.6.2 >Reporter: marco polo >Assignee: Eric Newton > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When running many batch scanners in a JVM, we see a significant amount of > cache invalidation within the JVM when we prematurely close the batch > scanner. When we prematurely close the batch scanner ( and threads within it > are running an interrupt is sent to those threads, causing them to add the > extents to the failures map and invalidate the cache. This causes lock > contention on the write lock for the TabletLocatorImpl. > The lock contention hinders performance in a highly parallel client that does > not need to invalid the cache as a result of being stopped. > As a positive test to ensure this was my problem I added a conditional check > to the IOException handler in the run method of > TabletServerBatchReaderIterator. The conditional checked whether the query > thread pool was shut down. If it was not, we would invalidate the cache as we > can assume the reason likely wasn't interruption. If it was shut down, we > would not invalide the cache. This reduced lock contention dramatically and > reduced runtime. This should cause no harm as any other failure would cause > cache invalidation through some other route. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-3835) TabletServerBatchReaderIterator concurrency contention on early close()
[ https://issues.apache.org/jira/browse/ACCUMULO-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060680#comment-15060680 ] Christopher Tubbs commented on ACCUMULO-3835: - Do we want to backport this bugfix to 1.6.5 or 1.7.1, since it was originally reported against 1.6.2? > TabletServerBatchReaderIterator concurrency contention on early close() > --- > > Key: ACCUMULO-3835 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3835 > Project: Accumulo > Issue Type: Bug > Components: client >Affects Versions: 1.6.2 >Reporter: marco polo >Assignee: Eric Newton > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When running many batch scanners in a JVM, we see a significant amount of > cache invalidation within the JVM when we prematurely close the batch > scanner. When we prematurely close the batch scanner ( and threads within it > are running an interrupt is sent to those threads, causing them to add the > extents to the failures map and invalidate the cache. This causes lock > contention on the write lock for the TabletLocatorImpl. > The lock contention hinders performance in a highly parallel client that does > not need to invalid the cache as a result of being stopped. > As a positive test to ensure this was my problem I added a conditional check > to the IOException handler in the run method of > TabletServerBatchReaderIterator. The conditional checked whether the query > thread pool was shut down. If it was not, we would invalidate the cache as we > can assume the reason likely wasn't interruption. If it was shut down, we > would not invalide the cache. This reduced lock contention dramatically and > reduced runtime. This should cause no harm as any other failure would cause > cache invalidation through some other route. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-3508) Zoocache utilizes course grained locking causing unnecessary synchronization in many tserver ops
[ https://issues.apache.org/jira/browse/ACCUMULO-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060689#comment-15060689 ] Christopher Tubbs commented on ACCUMULO-3508: - Should this be backported to 1.6.5 and/or 1.7.1, since it was originally reported against 1.6.0? > Zoocache utilizes course grained locking causing unnecessary synchronization > in many tserver ops > > > Key: ACCUMULO-3508 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3508 > Project: Accumulo > Issue Type: Sub-task > Components: fate >Affects Versions: 1.6.0 >Reporter: marco polo >Assignee: marco polo >Priority: Minor > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Through jstacking I have observed busy tablet server threads waiting for the > Zoocache to be updated or even read. Profiling via yourkit indicates > potential deadlocks when threads wait for a long period of time ( in my case > my waits were 10+s ) when many scans were running against my instance. > My patch, which is forthcoming, removes synchronization and utilizes a > reentrant read/write lock (RERWL) to synchronize the maps. My initial design > utilized concurrent hash maps, but the design of existence becomes simpler > and changes fewer by using the RERWL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (ACCUMULO-4076) Remove generated javadocs from docs/
[ https://issues.apache.org/jira/browse/ACCUMULO-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs reassigned ACCUMULO-4076: --- Assignee: Christopher Tubbs (was: Josh Elser) > Remove generated javadocs from docs/ > > > Key: ACCUMULO-4076 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4076 > Project: Accumulo > Issue Type: Sub-task > Components: build >Reporter: Josh Elser >Assignee: Christopher Tubbs > Fix For: 1.6.5, 1.7.1, 1.8.0 > > > In running some preliminary Yetus-driven tests, it was found (with the help > of [~aw]!) that we still dump the generate javadocs into docs/ instead of > something like docs/target. This caused Yetus to get a bit confused when > computing the effective diff of some patch being tested as suddenly there > were tons of changes from the original repository. > We stopped bundling the javadocs in our tarballs a while back (which, in > talking with [~ctubbsii] on IRC) is probably why they went to this > non-standard location. > We should make sure our generated javadocs are being put in a sane location, > likely the default location for the maven plugin itself is sufficient and > update the docs with any amended steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-4076) Remove generated javadocs from docs/
[
https://issues.apache.org/jira/browse/ACCUMULO-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060558#comment-15060558
]
Christopher Tubbs commented on ACCUMULO-4076:
-
Strange. It looks like the default location for {{javadoc:aggregate}} is
actually target/site/apidocs. *shrug*, that's fine with me.
> Remove generated javadocs from docs/
>
>
> Key: ACCUMULO-4076
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4076
> Project: Accumulo
> Issue Type: Sub-task
> Components: build
>Reporter: Josh Elser
>Assignee: Christopher Tubbs
> Fix For: 1.6.5, 1.7.1, 1.8.0
>
>
> In running some preliminary Yetus-driven tests, it was found (with the help
> of [~aw]!) that we still dump the generate javadocs into docs/ instead of
> something like docs/target. This caused Yetus to get a bit confused when
> computing the effective diff of some patch being tested as suddenly there
> were tons of changes from the original repository.
> We stopped bundling the javadocs in our tarballs a while back (which, in
> talking with [~ctubbsii] on IRC) is probably why they went to this
> non-standard location.
> We should make sure our generated javadocs are being put in a sane location,
> likely the default location for the maven plugin itself is sufficient and
> update the docs with any amended steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ACCUMULO-4078) Exclude special volumes
[ https://issues.apache.org/jira/browse/ACCUMULO-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Peterson updated ACCUMULO-4078: Description: A few improvements to the VolumeChooser are desired for a use case that seems general enough to warrant an update to Accumulo. Use Case: A new volume is added, with limited capacity, to be dedicated for a specific table. All other tables or files should be excluded from this new volume. Suggested Improvements: 1. Update the signature for VolumeManager.choose to take a VolumeChooserEnvironment instead of Optional. This will allow future parameters for volume selection without repeatedly changing the VolumeManager interface. 2. It's not currently possible to specify preferred volumes for the write-ahead logs 3. In several places including PreferredVolumeChooser, PerTableVolumeChooser and VolumeManagerImpl, the failsafe chooser is the RandomVolumeChooser which will include the instance volume that needs to be excluded. It would be useful to have a configurable failsafe in this situation. 4. The volume chooser is called in FileUtils for temp directory creation but it could instead use the sticky dir to create the temp directory, not needing the volume chooser at all. The above suggestions could become sub-tickets. Note that the improvements listed have been implemented for my particular instance and I hope to submit them as a patch. was: A few improvements to the VolumeChooser are desired for a particular use case that seems to be general enough that it warrants an updated to Accumulo. Perhaps these ideas should become sub-tickets. Note that the improvements listed below have been implemented for my particular instance and I expect to submit them as a patch. Use Case: A new volume is added, with limited capacity, to be dedicated for a specific table. All other tables or files should be excluded from this new volume. Suggested Improvements: 1. Update the signature for VolumeManager.choose to take a VolumeChooserEnvironment instead of Optional. This will allow future parameters for volume selection without repeatedly changing the VolumeManager interface. 2. It's not currently possible to specify preferred volumes for the write-ahead logs 3. In several places including PreferredVolumeChooser, PerTableVolumeChooser and VolumeManagerImpl, the failsafe chooser is the RandomVolumeChooser which will include the instance volume that needs to be excluded. It would be useful to have a configurable failsafe in this situation. 4. The volume chooser is called in FileUtils for temp directory creation but it could instead use the sticky dir to create the temp directory, not needing the volume chooser at all. > Exclude special volumes > --- > > Key: ACCUMULO-4078 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4078 > Project: Accumulo > Issue Type: Improvement > Components: core >Affects Versions: 1.7.0 >Reporter: Matt Peterson >Priority: Minor > Labels: newbie > > A few improvements to the VolumeChooser are desired for a use case that seems > general enough to warrant an update to Accumulo. > Use Case: > A new volume is added, with limited capacity, to be dedicated for a specific > table. All other tables or files should be excluded from this new volume. > Suggested Improvements: > 1. Update the signature for VolumeManager.choose to take a > VolumeChooserEnvironment instead of Optional. This will allow future > parameters for volume selection without repeatedly changing the VolumeManager > interface. > 2. It's not currently possible to specify preferred volumes for the > write-ahead logs > 3. In several places including PreferredVolumeChooser, PerTableVolumeChooser > and VolumeManagerImpl, the failsafe chooser is the RandomVolumeChooser which > will include the instance volume that needs to be excluded. It would be > useful to have a configurable failsafe in this situation. > 4. The volume chooser is called in FileUtils for temp directory creation but > it could instead use the sticky dir to create the temp directory, not needing > the volume chooser at all. > The above suggestions could become sub-tickets. Note that the improvements > listed have been implemented for my particular instance and I hope to submit > them as a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (ACCUMULO-4077) upgrade to thrift 0.9.3
[ https://issues.apache.org/jira/browse/ACCUMULO-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Newton resolved ACCUMULO-4077. --- Resolution: Fixed > upgrade to thrift 0.9.3 > --- > > Key: ACCUMULO-4077 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4077 > Project: Accumulo > Issue Type: Improvement > Components: client, master, proxy, tserver >Reporter: Eric Newton >Assignee: Eric Newton >Priority: Minor > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ACCUMULO-4078) Exclude special volumes
Matt Peterson created ACCUMULO-4078: --- Summary: Exclude special volumes Key: ACCUMULO-4078 URL: https://issues.apache.org/jira/browse/ACCUMULO-4078 Project: Accumulo Issue Type: Improvement Components: core Affects Versions: 1.7.0 Reporter: Matt Peterson Priority: Minor A few improvements to the VolumeChooser are desired for a particular use case that seems to be general enough that it warrants an updated to Accumulo. Perhaps these ideas should become sub-tickets. Note that the improvements listed below have been implemented for my particular instance and I expect to submit them as a patch. Use Case: A new volume is added, with limited capacity, to be dedicated for a specific table. All other tables or files should be excluded from this new volume. Suggested Improvements: 1. Update the signature for VolumeManager.choose to take a VolumeChooserEnvironment instead of Optional. This will allow future parameters for volume selection without repeatedly changing the VolumeManager interface. 2. It's not currently possible to specify preferred volumes for the write-ahead logs 3. In several places including PreferredVolumeChooser, PerTableVolumeChooser and VolumeManagerImpl, the failsafe chooser is the RandomVolumeChooser which will include the instance volume that needs to be excluded. It would be useful to have a configurable failsafe in this situation. 4. The volume chooser is called in FileUtils for temp directory creation but it could instead use the sticky dir to create the temp directory, not needing the volume chooser at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ACCUMULO-4079) ThriftTransportPool closes cached connections too aggressively
Eric Newton created ACCUMULO-4079: - Summary: ThriftTransportPool closes cached connections too aggressively Key: ACCUMULO-4079 URL: https://issues.apache.org/jira/browse/ACCUMULO-4079 Project: Accumulo Issue Type: Improvement Components: client Reporter: Eric Newton 3 seconds is a little fast. [~dlmarion] do you know what timeout value you've been using in production systems? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-4079) ThriftTransportPool closes cached connections too aggressively
[ https://issues.apache.org/jira/browse/ACCUMULO-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061288#comment-15061288 ] Dave Marion commented on ACCUMULO-4079: --- See ACCUMULO-2069. Almost two years to the day :-). I don't think we did anything, but I will double check when I get back in to the office to see if we changed something internally. > ThriftTransportPool closes cached connections too aggressively > -- > > Key: ACCUMULO-4079 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4079 > Project: Accumulo > Issue Type: Improvement > Components: client >Reporter: Eric Newton > > 3 seconds is a little fast. > [~dlmarion] do you know what timeout value you've been using in production > systems? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (ACCUMULO-4079) ThriftTransportPool closes cached connections too aggressively
[ https://issues.apache.org/jira/browse/ACCUMULO-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Newton resolved ACCUMULO-4079. --- Resolution: Duplicate Thanks [~dlmarion]. Please just update ACCUMULO-2069, if you made changes. > ThriftTransportPool closes cached connections too aggressively > -- > > Key: ACCUMULO-4079 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4079 > Project: Accumulo > Issue Type: Improvement > Components: client >Reporter: Eric Newton > > 3 seconds is a little fast. > [~dlmarion] do you know what timeout value you've been using in production > systems? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
