Accumulo-1.8 - Build # 267 - Fixed
The Apache Jenkins build system has built Accumulo-1.8 (build #267) Status: Fixed Check console output at https://builds.apache.org/job/Accumulo-1.8/267/ to view the results.
Accumulo-Master - Build # 2263 - Fixed
The Apache Jenkins build system has built Accumulo-Master (build #2263) Status: Fixed Check console output at https://builds.apache.org/job/Accumulo-Master/2263/ to view the results.
[GitHub] joshelser commented on issue #332: ACCUMULO-XXXX Use the hadoop3 shaded jars
joshelser commented on issue #332: ACCUMULO- Use the hadoop3 shaded jars URL: https://github.com/apache/accumulo/pull/332#issuecomment-366145399 Need to revisit again. Got sidetracked by other stuff. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (ACCUMULO-4817) Update build plugins and parent POM (Apache 19)
[ https://issues.apache.org/jira/browse/ACCUMULO-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4817. - Resolution: Fixed > Update build plugins and parent POM (Apache 19) > --- > > Key: ACCUMULO-4817 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4817 > Project: Accumulo > Issue Type: Improvement > Components: build >Reporter: Christopher Tubbs >Assignee: Christopher Tubbs >Priority: Major > Fix For: 1.7.4, 1.9.0, 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update build tooling across current development branches. Update Apache > parent POM to version 19, and update plugins to latest versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] madrob commented on issue #332: ACCUMULO-XXXX Use the hadoop3 shaded jars
madrob commented on issue #332: ACCUMULO- Use the hadoop3 shaded jars URL: https://github.com/apache/accumulo/pull/332#issuecomment-366140118 @joshelser what's the state here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ACCUMULO-4819) Fix numerous warnings introduced by Connector improvements
Christopher Tubbs created ACCUMULO-4819: --- Summary: Fix numerous warnings introduced by Connector improvements Key: ACCUMULO-4819 URL: https://issues.apache.org/jira/browse/ACCUMULO-4819 Project: Accumulo Issue Type: Improvement Components: client Reporter: Christopher Tubbs Fix For: 2.0.0 The changes in ACCUMULO-4784 deprecated the ClientConfiguration class, and introduced nearly 900 warnings about continued internal use of our own deprecated old API code. These warnings should be cleaned up, in order to reduce the noise, so new warnings (potential bugs) can be more easily triaged instead of lost in the noise... and so we can start using our own code internally. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ACCUMULO-4818) Upgrade to checkstyle 8+
Christopher Tubbs created ACCUMULO-4818: --- Summary: Upgrade to checkstyle 8+ Key: ACCUMULO-4818 URL: https://issues.apache.org/jira/browse/ACCUMULO-4818 Project: Accumulo Issue Type: Improvement Components: build Reporter: Christopher Tubbs Assignee: Christopher Tubbs Fix For: 2.0.0 Working on for ACCUMULO-4817, I noticed that our checkstyle version could be updated. However, this will involve some changes to our checkstyle configuration, since some of that is no longer compatible with the latest version of checkstyle (8.8). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ACCUMULO-4817) Update build plugins and parent POM (Apache 19)
Christopher Tubbs created ACCUMULO-4817: --- Summary: Update build plugins and parent POM (Apache 19) Key: ACCUMULO-4817 URL: https://issues.apache.org/jira/browse/ACCUMULO-4817 Project: Accumulo Issue Type: Improvement Components: build Reporter: Christopher Tubbs Assignee: Christopher Tubbs Fix For: 1.7.4, 1.9.0, 2.0.0 Update build tooling across current development branches. Update Apache parent POM to version 19, and update plugins to latest versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4813) Accepting mapping file for bulk import
[ https://issues.apache.org/jira/browse/ACCUMULO-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366396#comment-16366396 ] Michael Hogue commented on ACCUMULO-4813: - Would this replace the existing BulkImport algorithm or would it be additive? If the former, should there be a way to say which BulkImport algorithm you want to use for a given set of RFiles? > Accepting mapping file for bulk import > -- > > Key: ACCUMULO-4813 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4813 > Project: Accumulo > Issue Type: Sub-task >Reporter: Keith Turner >Priority: Major > Fix For: 2.0.0 > > > During bulk import, inspecting files to determine where they go is expensive > and slow. In order to spread the cost, Accumulo has an internal mechanism to > spread the work of inspecting files to random tablet servers. Because this > internal process takes time and consumes resources on the cluster, users want > control over it. The best way to give this control may be to externalize it > by allowing bulk imports to have a mapping file. This mapping file would > specify the ranges where files should be loaded. If Accumulo provided API to > help produce this file, then that work could be done in Map Reduce or Spark. > This would give users all the control they want over when and where this > computation is done. This would naturally fit in the process used to create > the bulk files. > To make bulk import fast this mapping file should have the following > properties. > * Key in file is a range > * Value in file is a list of files > * Ranges are non overlapping > * File is sorted by range/key > * Has a mapping for every non-empty file in the bulk import directory. > If Accumulo provides APIs to do the following operation, then producing the > file could written as a map/reduce job. > * For a given rfile produce a list of row ranges where the file should be > loaded. These row ranges would be based on tablets. > * Merge row range,list of file pairs > * Serialize row range,list of files pairs > With a mapping file, the bulk import algorithm could be written as follows. > This could all be executed in the master with no need to run inspection task > on random tablet servers. > * Sanity check file > ** Ensure in sorted order > ** Ensure ranges are non-overlapping > ** Ensure each file in directory has at least one entry in file > ** Ensure all splits in the file exist in the table. > * Since file is sorted can do a merged read of file and metadata table, > looping over the following operations for each tablet until all files are > loaded. > ** Read the loaded files for the tablet > ** Read the files to load for the range > ** For any files not loaded, send an async load message to the tablet server > The above algorithm can just keep scanning the metadata table and sending > async load messages until the bulk import is complete. Since the load > messages are async, the bulk load could of a large number of files could > potentially be very fast. > The bulk load operation can easily handle the case of tablets splitting > during the operation by matching a single range in the file to multiple > tablets. However attempting to handle merges would be a lot more tricky. It > would probably be simplest to fail the operation if a merge is detected. The > nice thing is that this can be done in a very clean way. Once the bulk > import operation has the table lock, merges can not happen. So after getting > the table lock the bulk import operation can ensure all splits in the file > exist in the table. The operation can abort if the condition is not met > before doing any work. If this condition is not met, it indicates a merge > happened between generating the mapping file an doing the bulk import. > Hopefully the mapping file plus the algorithm that sends async load messages > can dramatically speed up bulk import operations. This may lessen the need > for other things like prioritizing bulk import. To measure this, it would be > very useful create a bulk import performance test that can create many files > with very little data and measure the time it takes load them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ACCUMULO-4816) DfsLogger.readHeaderAndReturnStream should return a Closeable
[ https://issues.apache.org/jira/browse/ACCUMULO-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs updated ACCUMULO-4816: Affects Version/s: (was: 2.0.0) > DfsLogger.readHeaderAndReturnStream should return a Closeable > - > > Key: ACCUMULO-4816 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4816 > Project: Accumulo > Issue Type: Improvement >Reporter: Christopher Tubbs >Priority: Major > Fix For: 2.0.0 > > > With the change in https://github.com/apache/accumulo/pull/369, it reveals > that there's a possible unclosed resource hidden in the object returned from > DfsLogger.readHeaderAndReturnStream. This should return something which is a > Closeable, so it can be used in a try-with-resources block and cleaned up > properly just in case the extra stream allocates additional resources which > need to be closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ACCUMULO-4816) DfsLogger.readHeaderAndReturnStream should return a Closeable
[ https://issues.apache.org/jira/browse/ACCUMULO-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs updated ACCUMULO-4816: Fix Version/s: 2.0.0 > DfsLogger.readHeaderAndReturnStream should return a Closeable > - > > Key: ACCUMULO-4816 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4816 > Project: Accumulo > Issue Type: Improvement >Reporter: Christopher Tubbs >Priority: Major > Fix For: 2.0.0 > > > With the change in https://github.com/apache/accumulo/pull/369, it reveals > that there's a possible unclosed resource hidden in the object returned from > DfsLogger.readHeaderAndReturnStream. This should return something which is a > Closeable, so it can be used in a try-with-resources block and cleaned up > properly just in case the extra stream allocates additional resources which > need to be closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4809) Session manager clean up can happen when lock held.
[ https://issues.apache.org/jira/browse/ACCUMULO-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365998#comment-16365998 ] Christopher Tubbs commented on ACCUMULO-4809: - [~kturner], should this issue be closed? The PR has been merged. > Session manager clean up can happen when lock held. > --- > > Key: ACCUMULO-4809 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4809 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Priority: Critical > Labels: pull-request-available > Fix For: 1.7.4, 1.9.0, 2.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > While working on [PR #382|https://github.com/apache/accumulo/pull/382] for > ACCUMULO-4782 I noticed a significant concurrency bug. Before #382 their was > a single lock for the session manager. The session manager will clean up idle > sessions. This clean up should happen outside the session manager lock, > because all tserver read/write operation use the session manger so it should > be responsive. > The bug is the following. > * Both getActiveScansPerTable() and getActiveScans() lock the session > manager and then lock idleSessions. See [SessionManager line > 233|https://github.com/apache/accumulo/blob/rel/1.7.3/server/tserver/src/main/java/org/apache/accumulo/tserver/session/SessionManager.java#L233] > > * The sweep() method locks idleSessions and does cleanup while this lock is > held. [See SessionManager > 200|https://github.com/apache/accumulo/blob/rel/1.7.3/server/tserver/src/main/java/org/apache/accumulo/tserver/session/SessionManager.java#L200] > > Therefore it is possible for getActiveScansPerTable() or getActiveScans() to > lock the session manager and then block trying to lock idleSessions while > cleanup is happening in sweep(). This will block all access to the session > manager while cleanup happens. > The changes in #382 will fix this for 1.9.0 and 2.0.0. However I Am not sure > about backporting #382 to 1.7. A more targeted fix could be made for 1.7 or > #382 could be backported. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4803) Remove automatic KEK creation if SecretKeyEncryptionStrategies cannot find KEK file
[ https://issues.apache.org/jira/browse/ACCUMULO-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365994#comment-16365994 ] Christopher Tubbs commented on ACCUMULO-4803: - The existing implementations don't really need to change to create better implementations. The existing ones can remain, as they probably have value for performance testing with crypto. > Remove automatic KEK creation if SecretKeyEncryptionStrategies cannot find > KEK file > --- > > Key: ACCUMULO-4803 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4803 > Project: Accumulo > Issue Type: Improvement >Reporter: Nick Felts >Assignee: Nick Felts >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ACCUMULO-4792) Improve crypto configuration checks
[ https://issues.apache.org/jira/browse/ACCUMULO-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4792. - Resolution: Fixed Closing this. I think it's done. PR was merged. > Improve crypto configuration checks > --- > > Key: ACCUMULO-4792 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4792 > Project: Accumulo > Issue Type: Improvement >Reporter: Nick Felts >Assignee: Nick Felts >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The crypto module class and secret key encryption strategy class should each > be checked to ensure that if one is set not-null, the other is also enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ACCUMULO-4641) Modify BlockCache interface to avoid race conditions
[
https://issues.apache.org/jira/browse/ACCUMULO-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keith Turner resolved ACCUMULO-4641.
Resolution: Fixed
> Modify BlockCache interface to avoid race conditions
>
>
> Key: ACCUMULO-4641
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4641
> Project: Accumulo
> Issue Type: Sub-task
>Reporter: Keith Turner
>Assignee: Keith Turner
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently the BlockCache interface has functions to get and put. Accumulo
> will try to get a block, if it does not exist load it, and then put it in the
> cache. This can lead to race conditions where multiple threads unnecessarily
> load the same block.
> I think it would be better to modify the block cache interface to only have a
> function like the following.
> {code:java}
> CacheEntry get(String blockName, BlockLoader loader)
> {code}
> BlockLoader represents a function that the cache can call if a block is not
> present. The cache implementation can attempt to handle load race conditions
> however it likes..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Accumulo-Pull-Requests - Build # 1049 - Fixed
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #1049) Status: Fixed Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/1049/ to view the results.
[jira] [Commented] (ACCUMULO-4788) Improve Thrift Transport pool
[ https://issues.apache.org/jira/browse/ACCUMULO-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365926#comment-16365926 ] Keith Turner commented on ACCUMULO-4788: I need to open a follow up issue before closing this out. > Improve Thrift Transport pool > - > > Key: ACCUMULO-4788 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4788 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Accumulo has a pool of recently opened connections to tablet servers. When > connecting to tablet servers, this pool is checked first. The pool is built > around a map of list. There are two problems with this pool : > * It has global lock around the map of list > * When trying to find a connection it does a linear search for a non > reserved connection (this is per tablet server) > Could possibly move to a model of having a list of unreserved connections and > a set of reserved connections per tablet server. Then to get a connection, > could remove from the unreserved list and add to the reserved set. This > would be a constant time operation. > For the locking, could move to a model of using a concurrent map and locking > per tserver instead of locking the entire map. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4805) Seeing thread contention on FileManager
[ https://issues.apache.org/jira/browse/ACCUMULO-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365918#comment-16365918 ] Keith Turner commented on ACCUMULO-4805: I still want to create a follow on issue about lock contention and do a few things to do less work while the lock is held/ > Seeing thread contention on FileManager > --- > > Key: ACCUMULO-4805 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4805 > Project: Accumulo > Issue Type: Bug >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Accumulo has a tablet server wide cache of open files. Accessing this cache > obtains a global lock. In profiling, I am seeing contention on this lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ACCUMULO-4805) Seeing thread contention on FileManager
[ https://issues.apache.org/jira/browse/ACCUMULO-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357765#comment-16357765 ] Keith Turner edited comment on ACCUMULO-4805 at 2/15/18 4:55 PM: - The changes I made in #380 do not completely remove the contention, they just lessened the amount of time spent doing work while the lock is still held. was (Author: kturner): The changes I made in #380 do not completely remove the contention, they just lessened the amount of content. > Seeing thread contention on FileManager > --- > > Key: ACCUMULO-4805 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4805 > Project: Accumulo > Issue Type: Bug >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Accumulo has a tablet server wide cache of open files. Accessing this cache > obtains a global lock. In profiling, I am seeing contention on this lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ACCUMULO-4788) Improve Thrift Transport pool
[ https://issues.apache.org/jira/browse/ACCUMULO-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner updated ACCUMULO-4788: --- Fix Version/s: 1.9.0 > Improve Thrift Transport pool > - > > Key: ACCUMULO-4788 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4788 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Accumulo has a pool of recently opened connections to tablet servers. When > connecting to tablet servers, this pool is checked first. The pool is built > around a map of list. There are two problems with this pool : > * It has global lock around the map of list > * When trying to find a connection it does a linear search for a non > reserved connection (this is per tablet server) > Could possibly move to a model of having a list of unreserved connections and > a set of reserved connections per tablet server. Then to get a connection, > could remove from the unreserved list and add to the reserved set. This > would be a constant time operation. > For the locking, could move to a model of using a concurrent map and locking > per tserver instead of locking the entire map. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ACCUMULO-4801) Consider precomputing some client context fields
[ https://issues.apache.org/jira/browse/ACCUMULO-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner resolved ACCUMULO-4801. Resolution: Fixed > Consider precomputing some client context fields > > > Key: ACCUMULO-4801 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4801 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently each time a connection is requested from the the thrift transport > pool, three methods are called on client context to get ssl, sasl, and > timeout. These in turn call methods on configuration. This is showing up in > profiling as slow. I wonder if these could be precomputed in the client > context constructor. > > Also, repeatedly calling rpcCreds() on client context is showing up as slow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ACCUMULO-4801) Consider precomputing some client context fields
[ https://issues.apache.org/jira/browse/ACCUMULO-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner updated ACCUMULO-4801: --- Fix Version/s: 1.9.0 > Consider precomputing some client context fields > > > Key: ACCUMULO-4801 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4801 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently each time a connection is requested from the the thrift transport > pool, three methods are called on client context to get ssl, sasl, and > timeout. These in turn call methods on configuration. This is showing up in > profiling as slow. I wonder if these could be precomputed in the client > context constructor. > > Also, repeatedly calling rpcCreds() on client context is showing up as slow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ACCUMULO-4789) Scans spend significant time constructing debug string.
[ https://issues.apache.org/jira/browse/ACCUMULO-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner updated ACCUMULO-4789: --- Affects Version/s: 1.7.3 1.8.1 > Scans spend significant time constructing debug string. > --- > > Key: ACCUMULO-4789 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4789 > Project: Accumulo > Issue Type: Improvement >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > While profiling a Fluo test running lots of little scans, I noticed a string > builder operation showing up prominently in the profiling results. Below is > a link to the problematic code. Calling range toString was the most > expensive part followed by KeyExtent toString. > [https://github.com/apache/accumulo/blob/rel/1.7.3/core/src/main/java/org/apache/accumulo/core/client/impl/ThriftScanner.java#L405] > > I am not sure if we can change this in 1.7 and 1.8/1.9 because people may > rely on this for debugging. In 2.0 we may want to consider removing this (or > moving it inside the logging code block). > Also, while looking at this I noticed that some of the log statements called > String.format. Those should be placed in a if(llog.traceEnabled()) block. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ACCUMULO-4789) Scans spend significant time constructing debug string.
[ https://issues.apache.org/jira/browse/ACCUMULO-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner updated ACCUMULO-4789: --- Fix Version/s: 2.0.0 1.9.0 > Scans spend significant time constructing debug string. > --- > > Key: ACCUMULO-4789 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4789 > Project: Accumulo > Issue Type: Improvement >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > While profiling a Fluo test running lots of little scans, I noticed a string > builder operation showing up prominently in the profiling results. Below is > a link to the problematic code. Calling range toString was the most > expensive part followed by KeyExtent toString. > [https://github.com/apache/accumulo/blob/rel/1.7.3/core/src/main/java/org/apache/accumulo/core/client/impl/ThriftScanner.java#L405] > > I am not sure if we can change this in 1.7 and 1.8/1.9 because people may > rely on this for debugging. In 2.0 we may want to consider removing this (or > moving it inside the logging code block). > Also, while looking at this I noticed that some of the log statements called > String.format. Those should be placed in a if(llog.traceEnabled()) block. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ACCUMULO-4801) Consider precomputing some client context fields
[ https://issues.apache.org/jira/browse/ACCUMULO-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner reassigned ACCUMULO-4801: -- Assignee: Keith Turner > Consider precomputing some client context fields > > > Key: ACCUMULO-4801 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4801 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently each time a connection is requested from the the thrift transport > pool, three methods are called on client context to get ssl, sasl, and > timeout. These in turn call methods on configuration. This is showing up in > profiling as slow. I wonder if these could be precomputed in the client > context constructor. > > Also, repeatedly calling rpcCreds() on client context is showing up as slow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ACCUMULO-4789) Scans spend significant time constructing debug string.
[ https://issues.apache.org/jira/browse/ACCUMULO-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner reassigned ACCUMULO-4789: -- Assignee: Keith Turner > Scans spend significant time constructing debug string. > --- > > Key: ACCUMULO-4789 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4789 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > While profiling a Fluo test running lots of little scans, I noticed a string > builder operation showing up prominently in the profiling results. Below is > a link to the problematic code. Calling range toString was the most > expensive part followed by KeyExtent toString. > [https://github.com/apache/accumulo/blob/rel/1.7.3/core/src/main/java/org/apache/accumulo/core/client/impl/ThriftScanner.java#L405] > > I am not sure if we can change this in 1.7 and 1.8/1.9 because people may > rely on this for debugging. In 2.0 we may want to consider removing this (or > moving it inside the logging code block). > Also, while looking at this I noticed that some of the log statements called > String.format. Those should be placed in a if(llog.traceEnabled()) block. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ACCUMULO-4789) Scans spend significant time constructing debug string.
[ https://issues.apache.org/jira/browse/ACCUMULO-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner resolved ACCUMULO-4789. Resolution: Fixed > Scans spend significant time constructing debug string. > --- > > Key: ACCUMULO-4789 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4789 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > While profiling a Fluo test running lots of little scans, I noticed a string > builder operation showing up prominently in the profiling results. Below is > a link to the problematic code. Calling range toString was the most > expensive part followed by KeyExtent toString. > [https://github.com/apache/accumulo/blob/rel/1.7.3/core/src/main/java/org/apache/accumulo/core/client/impl/ThriftScanner.java#L405] > > I am not sure if we can change this in 1.7 and 1.8/1.9 because people may > rely on this for debugging. In 2.0 we may want to consider removing this (or > moving it inside the logging code block). > Also, while looking at this I noticed that some of the log statements called > String.format. Those should be placed in a if(llog.traceEnabled()) block. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4806) Allow offline bulk imports
[ https://issues.apache.org/jira/browse/ACCUMULO-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365891#comment-16365891 ] Mark Owens commented on ACCUMULO-4806: -- [~etcoleman] There is some discussion on modifying the way bulk imports work in general. The proposed change is described in ticket [ACCUMULO-4813|https://issues.apache.org/jira/browse/ACCUMULO-4813]. Can you read over that ticket and provide some thoughts from your perspective? This would provide the users control in how bulk imports are done. It is thought that if this was pursued it would negate the need for some of the other recently created bulk import sub-tasks. If the bulk import process time could be drop to under a minute, for example, the need for pause/resume would not really be needed, etc. > Allow offline bulk imports > -- > > Key: ACCUMULO-4806 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4806 > Project: Accumulo > Issue Type: Sub-task > Components: master, tserver >Reporter: Mark Owens >Assignee: Michael Miller >Priority: Major > Fix For: 2.0.0 > > > Allowing offline bulk imports would be useful for some customers. Currently > these customers already take tables offline to set split points but then have > to bring them back online before starting the import. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4788) Improve Thrift Transport pool
[ https://issues.apache.org/jira/browse/ACCUMULO-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365802#comment-16365802 ] Keith Turner commented on ACCUMULO-4788: In PR #385 the global lock was not addressed. The only change it made was to make the operations done when the lock was held much faster. > Improve Thrift Transport pool > - > > Key: ACCUMULO-4788 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4788 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Accumulo has a pool of recently opened connections to tablet servers. When > connecting to tablet servers, this pool is checked first. The pool is built > around a map of list. There are two problems with this pool : > * It has global lock around the map of list > * When trying to find a connection it does a linear search for a non > reserved connection (this is per tablet server) > Could possibly move to a model of having a list of unreserved connections and > a set of reserved connections per tablet server. Then to get a connection, > could remove from the unreserved list and add to the reserved set. This > would be a constant time operation. > For the locking, could move to a model of using a concurrent map and locking > per tserver instead of locking the entire map. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ACCUMULO-4782) With many threads scanning seeing lock contention on SessionManager
[ https://issues.apache.org/jira/browse/ACCUMULO-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner updated ACCUMULO-4782: --- Fix Version/s: 2.0.0 1.9.0 > With many threads scanning seeing lock contention on SessionManager > --- > > Key: ACCUMULO-4782 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4782 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > While profiling many threads doing small scans against accumulo, lock > contention on the tablet servers SessionManager was high. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ACCUMULO-4782) With many threads scanning seeing lock contention on SessionManager
[ https://issues.apache.org/jira/browse/ACCUMULO-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner reassigned ACCUMULO-4782: -- Assignee: Keith Turner > With many threads scanning seeing lock contention on SessionManager > --- > > Key: ACCUMULO-4782 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4782 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > While profiling many threads doing small scans against accumulo, lock > contention on the tablet servers SessionManager was high. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ACCUMULO-4782) With many threads scanning seeing lock contention on SessionManager
[ https://issues.apache.org/jira/browse/ACCUMULO-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner resolved ACCUMULO-4782. Resolution: Fixed > With many threads scanning seeing lock contention on SessionManager > --- > > Key: ACCUMULO-4782 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4782 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > While profiling many threads doing small scans against accumulo, lock > contention on the tablet servers SessionManager was high. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ACCUMULO-4798) Copying Stat in ZooCache is slow
[ https://issues.apache.org/jira/browse/ACCUMULO-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner reassigned ACCUMULO-4798: -- Assignee: Keith Turner > Copying Stat in ZooCache is slow > > > Key: ACCUMULO-4798 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4798 > Project: Accumulo > Issue Type: Improvement >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The ZooKeeper cache code caches Zookeeper stats. When a stat is requested > from the cache it copies it. The ZK stat class offers no good way to copy > other than serialize and deserialize. The code currently does this and its > slow. All code in Accumulo only uses one field from stat, so it would be > much better to create a simple class that has this one field and can quickly > copy. > > The stat is used very frequently in the metadata cache code to check if a > tserver still holds its lock. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ACCUMULO-4798) Copying Stat in ZooCache is slow
[ https://issues.apache.org/jira/browse/ACCUMULO-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Turner resolved ACCUMULO-4798. Resolution: Fixed > Copying Stat in ZooCache is slow > > > Key: ACCUMULO-4798 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4798 > Project: Accumulo > Issue Type: Improvement >Affects Versions: 1.7.3, 1.8.1 >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0, 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The ZooKeeper cache code caches Zookeeper stats. When a stat is requested > from the cache it copies it. The ZK stat class offers no good way to copy > other than serialize and deserialize. The code currently does this and its > slow. All code in Accumulo only uses one field from stat, so it would be > much better to create a simple class that has this one field and can quickly > copy. > > The stat is used very frequently in the metadata cache code to check if a > tserver still holds its lock. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4806) Allow offline bulk imports
[
https://issues.apache.org/jira/browse/ACCUMULO-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365467#comment-16365467
]
Ed Coleman commented on ACCUMULO-4806:
--
[~kturner] - Performing the import online, or offline as being discussed does
not really matter - but how would import errors work if the import was done
off-line? Would you still end up with files in the error directory when as part
of the bulk import - or after the table is brought online? {color:#33} What
is the failure reporting mechanism?{color}
> Allow offline bulk imports
> --
>
> Key: ACCUMULO-4806
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4806
> Project: Accumulo
> Issue Type: Sub-task
> Components: master, tserver
>Reporter: Mark Owens
>Assignee: Michael Miller
>Priority: Major
> Fix For: 2.0.0
>
>
> Allowing offline bulk imports would be useful for some customers. Currently
> these customers already take tables offline to set split points but then have
> to bring them back online before starting the import.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (ACCUMULO-4806) Allow offline bulk imports
[ https://issues.apache.org/jira/browse/ACCUMULO-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365452#comment-16365452 ] Ed Coleman edited comment on ACCUMULO-4806 at 2/15/18 12:26 PM: [~milleruntime] - It would save time of the total import by doing 1 and 2 were done before the RFiles were ready - this also applies to the current steps 1 - 4. Part of this issue for me is that the tserver seems to be doing a lot of work to offline the tablets. The other part is that it seems to be a cleaner API if there was something like: client.importWithSplits(tablename, import_dir, error_dir, splits) With the current process, this would take longer than may be necessary. was (Author: etcoleman): [~milleruntime] - It would save time of the total import by doing 1 and 2 were done before the RFiles were ready - this also applies to the current steps 1 - 4. Part of this issue for me is that the tserver seems to be doing a lot of work to offline the tablets. > Allow offline bulk imports > -- > > Key: ACCUMULO-4806 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4806 > Project: Accumulo > Issue Type: Sub-task > Components: master, tserver >Reporter: Mark Owens >Assignee: Michael Miller >Priority: Major > Fix For: 2.0.0 > > > Allowing offline bulk imports would be useful for some customers. Currently > these customers already take tables offline to set split points but then have > to bring them back online before starting the import. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ACCUMULO-4806) Allow offline bulk imports
[ https://issues.apache.org/jira/browse/ACCUMULO-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365452#comment-16365452 ] Ed Coleman commented on ACCUMULO-4806: -- [~milleruntime] - It would save time of the total import by doing 1 and 2 were done before the RFiles were ready - this also applies to the current steps 1 - 4. Part of this issue for me is that the tserver seems to be doing a lot of work to offline the tablets. > Allow offline bulk imports > -- > > Key: ACCUMULO-4806 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4806 > Project: Accumulo > Issue Type: Sub-task > Components: master, tserver >Reporter: Mark Owens >Assignee: Michael Miller >Priority: Major > Fix For: 2.0.0 > > > Allowing offline bulk imports would be useful for some customers. Currently > these customers already take tables offline to set split points but then have > to bring them back online before starting the import. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
