Here is the resource leak problem that we have encountered when 0.8.2 java
KafkaProducer failed in constructor. here is the code snippet of
KafkaProducer to illustrate the problem.
-------------------------------
public KafkaProducer(ProducerConfig config, Serializer<K> keySerializer,
Serializer<V> valueSerializer) {
// create metrcis reporter via reflection
List<MetricsReporter> reporters =
config.getConfiguredInstances(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG,
MetricsReporter.class);
// validate bootstrap servers
List<InetSocketAddress> addresses =
ClientUtils.parseAndValidateAddresses(config.getList(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG));
}
-------------------------------
let's say MyMetricsReporter creates a thread in constructor. if hostname
validation threw an exception, constructor won't call the close method of
MyMetricsReporter to clean up the resource. as a result, we created thread
leak issue. this becomes worse when we try to auto recovery (i.e. keep
creating KafkaProducer again -> failing again -> more thread leaks).
there are multiple options of fixing this.
1) just move the hostname validation to the beginning. but this is only fix
one symtom. it didn't fix the fundamental problem. what if some other lines
throw an exception.
2) use try-catch. in the catch section, try to call close methods for any
non-null objects constructed so far.
3) explicitly declare the dependency in the constructor. this way, when
KafkaProducer threw an exception, I can call close method of metrics
reporters for releasing resources.
KafkaProducer(..., List<MetricsReporter> reporters)
we don't have to dependency injection framework. but generally hiding
dependency is a bad coding practice. it is also hard to plug in mocks for
dependencies. this is probably the most intrusive change.
I am willing to submit a patch. but like to hear your opinions on how we
should fix the issue.
Thanks,
Steven
