[jira] [Updated] (KAFKA-3923) MetricReporter interface depends on final class KafkaMetric instead of Metric interface
[ https://issues.apache.org/jira/browse/KAFKA-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Stepanov updated KAFKA-3923: - Description: Hello, I'm working on exposing Kafka's consumer/producer metrics to Spring Actuator. To achieve this, I've implemented Kafka's MetricReporter interface to allow injecting it into the appropriate consumer/producer. No issues with implementation itself, fine for me. But now I've moved to writing unit tests for this implementation and decided to use mocked KafkaMetric instances for this. But mocking of KafkaMetric itself is not so plain - the class is final. The logical step is to use Metric interface for mocking, but MetricReporter accepts only KafkaMetric. I know that technically I can use PowerMock and most probably it will work fine in this case, but talking about Kafka itself, is it a good approach to depend on the exact implementation when interface is available? was: Hello, I'm working on exposing Kafka's consumer/producer metrics to Spring Actuator. To achieve this, I've implemented Kafka's MetricReporter interface to allow injecting it into the appropriate consumer/producer. No issues with implementation itself, fine for me. But now I've moved to writing unit tests for this implementation and decided to use mocked KafkaMetric instances for this. But mocking of KafkaMetric itself is not so plain - the class is final. The logical step is to use Metric interface for mocking, but MetricReporter accepts only KafkaMetric. I know that technically I can use PowerMock and most probably it will work fine in this case, but talking about Kafka itself, is it a good approach to depend on the exact implementation when interface fulfils most of the needs? > MetricReporter interface depends on final class KafkaMetric instead of Metric > interface > --- > > Key: KAFKA-3923 > URL: https://issues.apache.org/jira/browse/KAFKA-3923 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Igor Stepanov > > Hello, > I'm working on exposing Kafka's consumer/producer metrics to Spring Actuator. > To achieve this, I've implemented Kafka's MetricReporter interface to allow > injecting it into the appropriate consumer/producer. No issues with > implementation itself, fine for me. > But now I've moved to writing unit tests for this implementation and decided > to use mocked KafkaMetric instances for this. But mocking of KafkaMetric > itself is not so plain - the class is final. The logical step is to use > Metric interface for mocking, but MetricReporter accepts only KafkaMetric. > I know that technically I can use PowerMock and most probably it will work > fine in this case, but talking about Kafka itself, is it a good approach to > depend on the exact implementation when interface is available? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3923) MetricReporter interface depends on final class KafkaMetric instead of Metric interface
[
https://issues.apache.org/jira/browse/KAFKA-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358761#comment-15358761
]
Igor Stepanov commented on KAFKA-3923:
--
If you consider this worth to be improved, let me share what I've found already.
1. Tried to "generify" MetricsReporter:
{code:java}
public interface MetricsReporter extends Configurable {
public void init(List metrics);
public void metricChange(T metric);
...
}
{code}
This approach affected only MetricsReporter & JmxReporter and looked very
promising, but I've got an error in Scala code:
{quote}
/Users/igorstepanov/git/github/kafka/core/src/main/scala/kafka/server/KafkaConfig.scala:887:
trait MetricsReporter takes type parameters
val metricReporterClasses: java.util.List[MetricsReporter] =
getConfiguredInstances(KafkaConfig.MetricReporterClassesProp,
classOf[MetricsReporter])
^
/Users/igorstepanov/git/github/kafka/core/src/main/scala/kafka/server/KafkaConfig.scala:887:
trait MetricsReporter takes type parameters
val metricReporterClasses: java.util.List[MetricsReporter] =
getConfiguredInstances(KafkaConfig.MetricReporterClassesProp,
classOf[MetricsReporter])
^
/Users/igorstepanov/git/github/kafka/core/src/main/scala/kafka/server/KafkaServer.scala:99:
trait MetricsReporter takes type parameters
private val reporters: java.util.List[MetricsReporter] =
config.metricReporterClasses
{quote}
2. Tried just replacing KafkaMetric with Metric in most cases - this affected
also Metrics class (in addition to MetricsReporter & JmxReporter), but again
got error in Scala:
{quote}
/Users/igorstepanov/git/github/kafka/core/src/main/scala/kafka/server/ClientQuotaManager.scala:121:
type mismatch;
found : org.apache.kafka.common.Metric
required: org.apache.kafka.common.metrics.KafkaMetric
throttleTimeMs = throttleTime(clientMetric,
getQuotaMetricConfig(quota(clientId)))
{quote}
As I don't have the required experience with Scala, I've decided to stop at
this point and consult with you.
> MetricReporter interface depends on final class KafkaMetric instead of Metric
> interface
> ---
>
> Key: KAFKA-3923
> URL: https://issues.apache.org/jira/browse/KAFKA-3923
> Project: Kafka
> Issue Type: Improvement
> Components: clients
>Reporter: Igor Stepanov
>
> Hello,
> I'm working on exposing Kafka's consumer/producer metrics to Spring Actuator.
> To achieve this, I've implemented Kafka's MetricReporter interface to allow
> injecting it into the appropriate consumer/producer. No issues with
> implementation itself, fine for me.
> But now I've moved to writing unit tests for this implementation and decided
> to use mocked KafkaMetric instances for this. But mocking of KafkaMetric
> itself is not so plain - the class is final. The logical step is to use
> Metric interface for mocking, but MetricReporter accepts only KafkaMetric.
> I know that technically I can use PowerMock and most probably it will work
> fine in this case, but talking about Kafka itself, is it a good approach to
> depend on the exact implementation when interface fulfils most of the needs?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-3923) MetricReporter interface depends on final class KafkaMetric instead of Metric interface
Igor Stepanov created KAFKA-3923: Summary: MetricReporter interface depends on final class KafkaMetric instead of Metric interface Key: KAFKA-3923 URL: https://issues.apache.org/jira/browse/KAFKA-3923 Project: Kafka Issue Type: Improvement Components: clients Reporter: Igor Stepanov Hello, I'm working on exposing Kafka's consumer/producer metrics to Spring Actuator. To achieve this, I've implemented Kafka's MetricReporter interface to allow injecting it into the appropriate consumer/producer. No issues with implementation itself, fine for me. But now I've moved to writing unit tests for this implementation and decided to use mocked KafkaMetric instances for this. But mocking of KafkaMetric itself is not so plain - the class is final. The logical step is to use Metric interface for mocking, but MetricReporter accepts only KafkaMetric. I know that technically I can use PowerMock and most probably it will work fine in this case, but talking about Kafka itself, is it a good approach to depend on the exact implementation when interface fulfils most of the needs? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-3573) JSON (de)serialization for kafka-client
[ https://issues.apache.org/jira/browse/KAFKA-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Stepanov resolved KAFKA-3573. -- Resolution: Invalid > JSON (de)serialization for kafka-client > --- > > Key: KAFKA-3573 > URL: https://issues.apache.org/jira/browse/KAFKA-3573 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Igor Stepanov > > I've implemented generic implementations for: > - org.apache.kafka.common.serialization.Serializer > - org.apache.kafka.common.serialization.Deserializer > They provide support for serializing & deserializing POJO-entities into JSON > using: > - com.fasterxml.jackson.core:jackson-core > - com.fasterxml.jackson.core:jackson-databind > - org.apache.kafka.common.serialization.StringSerializer > - org.apache.kafka.common.serialization.StringDeserializer > The code is shared as a separate project on github: > https://github.com/stepio/kafka-json > But I may create a pull request for kafka-client to include this > implementation to the main code. What do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3573) JSON (de)serialization for kafka-client
[ https://issues.apache.org/jira/browse/KAFKA-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247641#comment-15247641 ] Igor Stepanov commented on KAFKA-3573: -- So the ticket is irrelevant for the project, right? If I got your point, you already have a JSON implementation and you'll work with it in the future. > JSON (de)serialization for kafka-client > --- > > Key: KAFKA-3573 > URL: https://issues.apache.org/jira/browse/KAFKA-3573 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Igor Stepanov > > I've implemented generic implementations for: > - org.apache.kafka.common.serialization.Serializer > - org.apache.kafka.common.serialization.Deserializer > They provide support for serializing & deserializing POJO-entities into JSON > using: > - com.fasterxml.jackson.core:jackson-core > - com.fasterxml.jackson.core:jackson-databind > - org.apache.kafka.common.serialization.StringSerializer > - org.apache.kafka.common.serialization.StringDeserializer > The code is shared as a separate project on github: > https://github.com/stepio/kafka-json > But I may create a pull request for kafka-client to include this > implementation to the main code. What do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3573) JSON (de)serialization for kafka-client
[ https://issues.apache.org/jira/browse/KAFKA-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244831#comment-15244831 ] Igor Stepanov commented on KAFKA-3573: -- Good point, found your code here: https://github.com/apache/kafka/tree/trunk/connect/json/src/main/java/org/apache/kafka/connect/json Existing implementation works with JsonNode, mine - with POJOs. You may check the tests for details. Are you interested in such an approach? P.S.: loved your idea of serializing directly to byte[] (avoiding the intermediate String), so implemented the same approach. > JSON (de)serialization for kafka-client > --- > > Key: KAFKA-3573 > URL: https://issues.apache.org/jira/browse/KAFKA-3573 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Igor Stepanov > > I've implemented generic implementations for: > - org.apache.kafka.common.serialization.Serializer > - org.apache.kafka.common.serialization.Deserializer > They provide support for serializing & deserializing POJO-entities into JSON > using: > - com.fasterxml.jackson.core:jackson-core > - com.fasterxml.jackson.core:jackson-databind > - org.apache.kafka.common.serialization.StringSerializer > - org.apache.kafka.common.serialization.StringDeserializer > The code is shared as a separate project on github: > https://github.com/stepio/kafka-json > But I may create a pull request for kafka-client to include this > implementation to the main code. What do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3573) JSON (de)serialization for kafka-client
Igor Stepanov created KAFKA-3573: Summary: JSON (de)serialization for kafka-client Key: KAFKA-3573 URL: https://issues.apache.org/jira/browse/KAFKA-3573 Project: Kafka Issue Type: New Feature Components: clients Reporter: Igor Stepanov I've implemented generic implementations for: - org.apache.kafka.common.serialization.Serializer - org.apache.kafka.common.serialization.Deserializer They provide support for serializing & deserializing POJO-entities into JSON using: - com.fasterxml.jackson.core:jackson-core - com.fasterxml.jackson.core:jackson-databind - org.apache.kafka.common.serialization.StringSerializer - org.apache.kafka.common.serialization.StringDeserializer The code is shared as a separate project on github: https://github.com/stepio/kafka-json But I may create a pull request for kafka-client to include this implementation to the main code. What do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
