[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-25 Thread Ted Yu (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4863:
--

Fix Version/s: 0.94.0

+1 on patch v2.
The test failures reported by HadoopQA weren't related to the patch.

> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 
> 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 
> 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, 
> D531.2.patch, D531.3.patch, D531.4.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-25 Thread Mikhail Bautin (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4863:
--

Status: Patch Available  (was: Open)

> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: 
> 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 
> 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, 
> D531.2.patch, D531.3.patch, D531.4.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-25 Thread Mikhail Bautin (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4863:
--

Attachment: 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch

> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: 
> 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 
> 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, 
> D531.2.patch, D531.3.patch, D531.4.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-25 Thread Mikhail Bautin (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4863:
--

Status: Open  (was: Patch Available)

> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: 
> 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 
> 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, 
> D531.2.patch, D531.3.patch, D531.4.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-25 Thread Phabricator (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4863:
---

Attachment: D531.4.patch

mbautin updated the revision "[jira] [HBASE-4863] Make HBase Thrift server more 
configurable and add a command-line UI test".
Reviewers: JIRA, Kannan, tedyu, stack

  Fixing a bug in TestThreads. Cluster testing is going well. I will kick off a 
unit test run on Jenkins.

REVISION DETAIL
  https://reviews.facebook.net/D531

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java
  src/main/java/org/apache/hadoop/hbase/util/Threads.java
  src/main/resources/hbase-default.xml
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
  src/test/java/org/apache/hadoop/hbase/util/TestThreads.java


> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: 
> 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 
> 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, 
> D531.2.patch, D531.3.patch, D531.4.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-24 Thread Mikhail Bautin (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4863:
--

Attachment: 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch

The same as D531.3.patch but generated using "git format-patch --no-prefix 
HEAD^..HEAD" so that it can be applied using the normal patch command.

> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: 
> 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, 
> D531.2.patch, D531.3.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-24 Thread Mikhail Bautin (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4863:
--

Status: Patch Available  (was: Open)

> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: 
> 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, 
> D531.2.patch, D531.3.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-24 Thread Phabricator (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4863:
---

Attachment: D531.3.patch

mbautin updated the revision "[jira] [HBASE-4863] Make HBase Thrift server more 
configurable and add a command-line UI test".
Reviewers: JIRA, Kannan, tedyu, stack

  Addressing Ted's comments. I will re-run unit tests and cluster tests, and 
post an update.

REVISION DETAIL
  https://reviews.facebook.net/D531

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java
  src/main/java/org/apache/hadoop/hbase/util/Threads.java
  src/main/resources/hbase-default.xml
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
  src/test/java/org/apache/hadoop/hbase/util/TestThreads.java


> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: D531.1.patch, D531.2.patch, D531.3.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-24 Thread Phabricator (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4863:
---

Attachment: D531.2.patch

mbautin updated the revision "[jira] [HBASE-4863] Make HBase Thrift server more 
configurable and add a command-line UI test".
Reviewers: JIRA, Kannan, tedyu, stack

  Updating with the most recent version. Posted a stale version at first -- 
sorry for spam.

REVISION DETAIL
  https://reviews.facebook.net/D531

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/thrift/HBaseThreadPoolServer.java
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java
  src/main/java/org/apache/hadoop/hbase/util/Threads.java
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
  src/test/java/org/apache/hadoop/hbase/util/TestThreads.java


> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: D531.1.patch, D531.2.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test

2011-11-24 Thread Phabricator (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4863:
---

Attachment: D531.1.patch

mbautin requested code review of "[jira] [HBASE-4863] Make HBase Thrift server 
more configurable and add a command-line UI test".
Reviewers: JIRA, Kannan, tedyu, stack

  This started as an internal hotfix where we found out that the Thrift server 
spawned 15000 threads. To bound the thread pool size I added a custom thread 
pool server implementation called HBaseThreadPoolServer into HBase codebase, 
and made the following parameters configurable from both command line and as 
config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
Under an increasing load, the server creates new threads for every connection 
before the pool size reaches minWorkerThreads. After that, the server puts new 
connections into the queue and only creates a new thread when the queue is 
full. If an attempt to create a new thread fails, the server drops connection. 
The default TThreadPoolServer would crash in that case, but it never happened 
because the thread pool was unbounded, so the server would hang indefinitely, 
consume a lot of memory, and cause huge latency spikes on the client side.

  Another part of this fix is refactoring and unit testing of the command-line 
part of the Thrift server. The logic there is sufficiently complicated, and the 
existing ThriftServer class does not test that part at all. The new 
TestThriftServerCmdLine test starts the Thrift server on a random port with 
various combinations of options and talks to it through the client API from 
another thread.


TEST PLAN
  Unit tests, cluster test with a Python Thrift client.
  I will post an update when I'm done with testing.

REVISION DETAIL
  https://reviews.facebook.net/D531

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/thrift/HBaseThreadPoolServer.java
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java
  src/main/java/org/apache/hadoop/hbase/util/Threads.java
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
  src/test/java/org/apache/hadoop/hbase/util/TestThreads.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/1167/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


> Make HBase Thrift server more configurable and add a command-line UI test
> -
>
> Key: HBASE-4863
> URL: https://issues.apache.org/jira/browse/HBASE-4863
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: D531.1.patch
>
>
> This started as an internal hotfix where we found out that the Thrift server 
> spawned 15000 threads. To bound the thread pool size I added a custom thread 
> pool server implementation called HBaseThreadPoolServer into HBase codebase, 
> and made the following parameters configurable from both command line and as 
> config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. 
> Under an increasing load, the server creates new threads for every connection 
> before the pool size reaches minWorkerThreads. After that, the server puts 
> new connections into the queue and only creates a new thread when the queue 
> is full. If an attempt to create a new thread fails, the server drops 
> connection. The default TThreadPoolServer would crash in that case, but it 
> never happened because the thread pool was unbounded, so the server would 
> hang indefinitely, consume a lot of memory, and cause huge latency spikes on 
> the client side.
> Another part of this fix is refactoring and unit testing of the command-line 
> part of the Thrift server. The logic there is sufficiently complicated, and 
> the existing ThriftServer class does not test that part at all. The new 
> TestThriftServerCmdLine test starts the Thrift server on a random port with 
> various combinations of options and talks to it through the client API from 
> another thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira