OpenCA testing Update (September 2004)

Testing environment

Hardware

Dell 2650 Server

2x2.8GHz processors,

1 Gbyte RAM,

70Bbyte RAID 1 disk)

Operating System

Fedora Core 2

OpenCA configuration

OpenCA 092, CVS

Single shared CA/RA database using mySQL.

Volumes

50,000 certificates (batch generated)

Batch-Processors

Summary

High-level comments on current batch-processor operation.

The batch tool provides a convenient method of generating certificates in volume as long as batch sizes are kept to a maximum of around 3000 certificates.

The batch tool suffers significant fall-off in performance as batch volume sizes increase, with rates of around 1600 users per hour being achieved with a batch size of 1000 falling to 950 users per hour at batch sizes of 3000.

The quantity of information sent back to the browser during a batch run is simply too great if any form of volume is run. It is quite common to the browser to become totally unstable and unresponsive.

It is believed that the fall-off in batch performance is related to excessive memory usage, which becomes apparent during the create_csr job step and then gets progressively worse. On the test system a 3000-batch run will show the OpenCA module consuming around 70% of the system memory.

The batch system generates significant number of error log entries (.XML files), around 40,000 for the standard 3000-user batch test. This effectively overloads the LOG system making it unusable.

In general the batch system does appear to be able to recovery from errors but on at least one occasion (failed part way through certificate issue job step) the batch process would not recover and causes a 4000 cert hole to be created in the certificate database.

It is recommended that a full CA backup is taken (DB and file system) before any large batch process is run as failure could make CA recovery a difficult activity.

The batch process effectively locks out any CA user looking at the certificate and request database tables.

The batch-process tmp files must be tidies up before the directory count exceeds 32000 as most Linux systems have this inbuilt limit for directories within a directory.

Specific errors

Performance/memory leak problem

The performance and memory use of the batch processor falls off significantly as volumes exceed 1000 users in a single batch. The problem appears to be most serious in job steps from create_csr onwards.

Failed unrecoverable batch

A batch run of 4000 items was run that failed midway through the certificate generation phase (some processed some not) due to a max links exceeded error.

The workflow function create_cert crashs for process keys of user GPcAOVLP_16997. mkdir /usr/local/volume_pki/openca/var/tmp/msg-1095497448-7241-3994: Too many links

mkdir /usr/local/volume_pki/openca/var/tmp/msg-1095497448-7241-3994: Too many links

0

The function create_cert for process keys of user GPcAOVLP_16997 succeeded.

In practice all certificates were issued and the certificate serial number counter incremented. The certificates can be found in the /var/crypto/certs directory but not in the database. The csr appears not to be present in the database either which you would have thought was impossible !

When the /var/tmp directory was cleared of files the batch was restarted but all users failed and no further progress could be made.

Subsequent batches ran OK.

Batch process does not tidy /var/tmp on error

When the batch process detects an error (I think I forced the file links error to cause this) the /var/tmp directory can be left containing various openssl_backup type files (index and serial files I think). It also leaves various temporary directories containing revocation email files in which actually make it difficult to know what directories are proper BP directories and which are those left over from a failure as the naming convention for both is very similar.

As an observation, it is always a concern to find files that have been created which contain all cert data as the openss_backup_*_index.txt file appears to. Obviously at low CA volumes this is not a problem but at higher volumes this is surely not at all desirable.

Comments

Database updates

When performing updates to the database entries (e.g. the CSR in the request table) to SQL query always sends a complete copy of the database row to the SQL server rather than just sending the changed item. This appears to be an inefficient way of doing things when quite often only the “state” element is being updated.

Logging

A re-think is needed regarding the logging of the batch-processor activity. Whilst it is appreciated that logging is an important element of all OpenCA activities it is difficult to see how 40,000 log entries for a 3000 user batch run is going to be of any use in the future. At present it is actually making OpenCA logging as a whole worse because the 40,000 log entries swamp the standard logs making access to them virtually impossible especially as the searching facilities are limited.

I think a different approach to logging in the batch processors is needed, possibly making logging configurable for specific “important” job steps, like cert creation. Another approach might be to allow the batch process to log to its own log file so that it doesn’t effect normal OpenCa logging and also archived easily and separately.

Batch processor clear down

The batch process creates a number of files and entries in various places. If you run 2 consecutive batch runs then these data entries can be difficult to separate one from another (pin.waiting, pkcs12.waiting, files held in /var/tmp, files held in /var/bp/users, etc).

It would be useful if the data from individual batch-process was better separated and a screen based method provided to clear-down specific runs.

Use of /var/tmp and /var/bp/dataexchange

I’m not quite sure why the batch-processors use a mix of /var/tmp, /var/tmp/msg-*/ and /var/bp/dataexchange to store the various user file outputs (some mails and some P12s).

Certainly you have to be careful clearing up the files from /var/tmp as this directory is used for various other tmp files (XML) which you might not pick up on given the volume of “??????????” files and “MSG-*” directories created there by BP.

Also at high-ish volumes you can no longer use “rm” to tidy the files as it breaks the parameter passing limit of bash/rm.

Logging

Summary

The existing log system is virtually unusable in a large volume CA as the time taken to search combined with the lack of searching flexibility make it impossible to track down the logs associated to any specific event (e.g. a specific user or logon).

The logging system with it’s individual .xml files within the nested directories and linked “all” areas appears to place a significant load on the system especially when performing backups.

It is suggested that the default logging for a new OpenCA install should be changed from the current log all options.

It is not clear in the manual how the configuration of the logging system is achieved.

Specific Errors

Index rebuild

Deletion of .DBM files

Comments

I know a lot of work is planned for OpenCA V093 but we need to ensure that OpenCA’s ability to log events doesn’t run ahead it’s ability to then use them as I suspect is the case now.

User screens

Summary

At the 50K volume the response to the general query screens (certificates and requests) can be slow initially (say 15 to 25 seconds) while the initial select count(*) query is processed. Subsequent page forward or page back links are generally completed within 2 or 3 seconds.

The RA facility to email users of certificates about to expire is unusable in its current form. Any significant OpenCA volumes take a huge amount of time to process.

The certificate search facility should be provided within the CA too as rising volumes make the general query screens less and less useful.

As the screen response time increases the length of time the user can be left in “limbo” after a screen link is pressed can be a bit disconcerting.

Specific Errors

Warn user of expiring certificate (RA function), performance problem

The warn user function takes approximately 30 minutes to run with a 10000 user CA which is far too long. I suspect the problem with the performance of the warm facility appears to be that a database call of the form:

select min(cert_key) from certificate where cert_key > last_cert_processed

and status like “valid” and notafter < datenow_plus_14_days;

is made for each certificate in the database, starting at the beginning. I would have though it possible to write a single select to return all certificates rather than retrieving the index for the next one that needs processing and then presumably making another call to actually get the certificate info (if required).

View Certificates “<<<<<” links can fail

I think the current “back-link” (links to move back through the list of valid certificates “<<<<”) code assumes that the database does not contain any missing serial numbers so it searches for the current lowest serial number displayed minus a number back depending upon which back link is pressed. In practice there will always be missing serial numbers as most of the screen queries are also looking for a specific state so in a live CA you could quite easily have a block of revoked certs that mess the back counting up. In my case the CA has 4000 “missing” certificates (caused by the BP failing), which makes it very difficult to move back through the certificate screens.

I have mainly checked looking at “live” certificates and “archived” requests but if the same “back link” code is used for less normal states then the problem is likely to be worse as, for example, revoked certs will be spread thinly right through the database.

I think a general SQL query of the following form might solve th problem:

Select * from certificate where (cert_key < current_min_displayed_cert_key) and
(state like “required_state”) and
(notafter > date_time_now) order by cert_ket desc limit 20;

Cert and request queries fail during batch process

It appears that the BP when running can lock out any CA user queries that require access to the certificate and request tables. Not quite sure what causes it but it appears that the “select count(*)” request issued at the beginning of each screen query/page view never gets a response.

I have also tries issuing the same “count (*)” query from the mySQL command line and get the same problem. No errors are reported, it just takes a long time, over 2 hours on one occasion.

Comments

The following general comments apply.

The use of the < > navigation keys at the bottom of each display page are fine but unfortunately they generally appear off the bottom of the page which means to use them you need to first scroll down and then press a link. I think it would be more usable if the links were accessible straight away (possibly at the top or both top and bottom).

The “range” of each link is not particularly intuitive. I can’t quite work out what he steps are but generally I think humans work better with nice round numbers like 50, 100,500,1000. It might also be useful to use the html title tag so the current jump value pops up when the user hovers over the link.

A number of OpenCA screens now use an on-load function to place the focus on the first action field in the page. Whilst I understand the reasoning behind this in practice it tends to push the important information off the top of the page, which requires the user to scroll up (to check they have the right item) and then scroll back down again to press the action link. Placing focus on a field also disables the window scroll so it is also necessary to click on the main frame and then scroll.

MySQL query cache introduced with V4 could well help screen performance as the count, min and max queries are repeated many times.

It would be good if the main display area is at least cleared immediately an action link is pressed, this way at least the user can see that their request is being dealt with. It’s the normal “please wait” type of approach.

System management

Backup/Recovery

At present, the 50K CA takes approximately 20 minutes to create a tar of the whole environment (database and file system) assuming that the LOG system has been cleared of all messages. Whilst this is OK for the time being there will come a time when the current backup time will start to cause problems.

A database only backup takes around 4 minutes.

The “many files” directories account for most of the time taken (e.g. .XML, certs, crins, mails). Possibly there is a case for archiving these files, many of which are never likely to be used.

OpenCA server process

Stopping an action

Now that OpenCA has a server component it is not possible for the user to stop a request for action from the browser. Whilst this may not be a major problem it might be an issue if a user accidentally selects a time consuming task. As far as the user is concerned they can actually follow other OpenCA links but the OpenCA server will continue to process the original request in the background.

Multiple OpenCA servers

The OpenCA server always has 2 entries in the running processes tables where it is more usual to find only a single copy of a resident process.

Database tuning

A brief investigation of database indexing reveals the following:

Adding indexes for public_key (50) (certificate & request tables) and req_key (certificate table) appears to offer a reasonable performance increase when using the batch processors. Elapsed time for a 1000 user test dropped from 1hour 17 minutes to 39 mins with indexes.

The addition of a state index (request table) and status/notafter index (certificate) table made little noticeable difference to batch or screen performance.

Indexes appear to make little difference to the overall response time of the user screens.

OpenCA testing Update

Hobbs

2004-09-27T12:26:00