GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/608
METRON-986 Enhance Fastcapa to Support Intel X520
### Changes
The following changes have been made to Fastcapa to improve
interoperability with the Intel X520 NIC, address bugs uncovered during
performance testing, and some basic enhancements to improve ease-of-use.
* Created a separate command-line parameter for the transmit worker burst
size; `-w TX_BURST_SIZE`. This defines the maximum number of packets dequeued
from the transmit ring and enqueued for delivery to Kafka at one time.
Previously the burst size for the receive and transmit worker were the same.
In practice the transmit worker burst size needs to be roughly 10x greater than
the receive worker burst size.
* Fixed a bug that caused some failures reported through the message
delivery callback to be ignored. Both successful and failed deliveries are
reported through this callback and the logic has been corrected to match this.
* Error messages for each failed message can be logged when the `DEBUG`
flag is used during compilation. By default errors for each message are not
logged due to performance implications.
* Now logging all configuration parameters by default.
* The code was unnecessarily limiting `-x TX_RING_SIZE` by using a
uint16_t; a 32-bit value is perfectly acceptable. Also, no need to validate
the -x TX_RING_SIZE as DPDK's EAL does this for us.
* Added separate default port configurations for the receive and transmit
side. Previously the same configuration values were used for both.
* Added logic to count, summarize and report all ethernet device errors.
* Refactored logic so that all receive and transmit worker logic is
contained within `worker.[c|h]` and all NIC logic contained within `nic.[c|h]`.
* Updated the fastcapa service script to allow the process to be stopped,
killed, and the logs tailed.
### Testing
This change has been tested on a live cluster with the Intel X520 NIC. The
probe was able to capture and land roughly 1 gbps of raw packet data into
Kafka. The change was also tested in the Fastcapa test environment.
To test the change yourself, simply spin-up the virtualized test
environment which will deploy and validate that Fastcapa can land packets in
Kafka correctly.
```
cd metron-deployment/vagrant/fastcapa-test-environment
vagrant up
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-986
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/608.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #608
----
commit 5559c00b3ad569a4b6aa3a48c89bfbb21f6d4830
Author: Nick Allen <[email protected]>
Date: 2017-05-05T18:16:49Z
METRON-986 Enhance Fastcapa to Support Intel X520
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
