We are seeing exceptions where Hbase REST is returning an invalid status code
on writes, causing our Python processes to error out on trying to make a write
(the following error is thrown by the Python starbase library, but is not
caused by starbase directly):
`requests.exceptions.ConnectionError: ('Connection aborted.',
BadStatusLine("''",))`
There are no traces nor errors reported in the REST log at the time of the
exception. We know that this is consistent, as we retry thrice with exponential
backoff (2 seconds initially).
Our REST threads are allowed to grow dynamically from 100 to 9000 at worst; our
connections typically do not exceed these values, and did not at the time.
Are there circumstances where Hbase REST will deliberately refuse to honor a
connection or respond with invalid codes? Or should I continue looking for more
environmental reasons? Note that at the time REST was documenting wait on a
number of AsyncProcesses on several tasks after having recently closed a table
- not sure how useful that information is. A log snippet at the time from Hbase
REST is below:
```
2017-02-07 17:07:35,312 INFO [196541187@qtp-1054818236-2084]
client.HBaseAdmin: Started disable of urls_table_50007_20170207170255
2017-02-07 17:07:43,294 INFO [196541187@qtp-1054818236-2084]
client.HBaseAdmin: Disabled urls_table_50007_20170207170255
2017-02-07 17:07:44,093 INFO [196541187@qtp-1054818236-2084]
client.HBaseAdmin: Deleted urls_table_50007_20170207170255
2017-02-07 17:07:53,238 INFO [115866407@qtp-1054818236-2339]
client.HBaseAdmin: Started disable of urls_table_50001_20170207170502
2017-02-07 17:07:59,953 INFO [115866407@qtp-1054818236-2339]
client.HBaseAdmin: Disabled urls_table_50001_20170207170502
2017-02-07 17:08:02,128 INFO [115866407@qtp-1054818236-2339]
client.HBaseAdmin: Deleted urls_table_50001_20170207170502
2017-02-07 17:08:04,585 INFO [328489809@qtp-1054818236-2207]
client.HBaseAdmin: Started disable of urls_table_50023_20170207170455
2017-02-07 17:08:04,791 INFO [1227614359@qtp-1054818236-647]
client.HBaseAdmin: Started disable of urls_table_50028_20170207170311
2017-02-07 17:08:05,664 INFO [1252180146@qtp-1054818236-143]
client.HBaseAdmin: Started disable of urls_table_50036_20170207170458
2017-02-07 17:08:07,783 INFO [328489809@qtp-1054818236-2207]
client.HBaseAdmin: Disabled urls_table_50023_20170207170455
2017-02-07 17:08:14,066 INFO [1474938506@qtp-1054818236-2246]
client.AsyncProcess: #1361530, waiting for some tasks to finish. Expected
max=0, tasksInProgress=7
2017-02-07 17:08:14,167 INFO [865578926@qtp-1054818236-1712]
client.AsyncProcess: #1361562, waiting for some tasks to finish. Expected
max=0, tasksInProgress=7
2017-02-07 17:08:14,267 INFO [1276865244@qtp-1054818236-1407]
client.AsyncProcess: #1361556, waiting for some tasks to finish. Expected
max=0, tasksInProgress=7
2017-02-07 17:08:14,267 INFO [1368235409@qtp-1054818236-315]
client.AsyncProcess: #1361671, waiting for some tasks to finish. Expected
max=0, tasksInProgress=7
```
Akshat
