[jira] [Created] (KAFKA-7463) ConsoleProducer rejects producer property values with equals signs
Magnus Reftel created KAFKA-7463:
Summary: ConsoleProducer rejects producer property values with
equals signs
Key: KAFKA-7463
URL: https://issues.apache.org/jira/browse/KAFKA-7463
Project: Kafka
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Magnus Reftel
The --producer-property option of ConsoleProducer rejects proprty values with
equals signs with the message "Invalid command line properties". This makes
specifying SASL configuration via the the sasl.jaas.config property difficult,
since these values often contain equals signs. It's possible to work around the
issue by writing the SASL configuration to a separate file, and then specifying
it via
{code:java}
KAFKA_OPTS=-Djava.security.auth.login.config=$filename{code}
but there seems to be no good reason to not allow the more direct route.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Dots in topic names in metric labels
Hi, Dots in topic names in labels are replaced with underscores in consumer metrics (it appears it has been that way all the way back since KAFKA-3233), but not in broker or producer metrics. We'd like to make visualizations that are parameterized by topic name, and show overview information about the whole chain of systems that interact with that topic, but this gets a bit tricky since we have dots in our topic names. Is there any particular reason that dots in topic names are replaced with underscores in consumer metrics labels? If not, would it be possible to change this, or would that be considered an incompatible change of an API? Best Regards Magnus Reftel Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan vaere belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.
[GitHub] kafka pull request #3267: MINOR: more details in error message descriptions
GitHub user reftel opened a pull request: https://github.com/apache/kafka/pull/3267 MINOR: more details in error message descriptions You can merge this pull request into a Git repository by running: $ git pull https://github.com/reftel/kafka feature/error_descriptions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3267.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3267 commit 01f9cf1fbba97f01de717adfda3b5980fe321ef4 Author: Magnus Reftel Date: 2017-06-08T08:28:38Z MINOR: more details in error message descriptions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[ https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040485#comment-16040485 ] Magnus Reftel commented on KAFKA-5390: -- Will do. Thanks! > First records in batch rejected but others accepted when rolling log > > > Key: KAFKA-5390 > URL: https://issues.apache.org/jira/browse/KAFKA-5390 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.2.1 >Reporter: Magnus Reftel > Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz > > > When sending a sequence of records in a batch right when the broker needs to > roll a new segment, it's possible for the first few records to fail, while > other records in the batch are accepted. If records have dependencies on > earlier records, e.g. in the case of a sequence of events in an event-sourced > system, then a producer cannot use the batching functionality, since it then > risks consumers receiving a record without first receiving the records it > depends on. > See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[ https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Reftel resolved KAFKA-5390. -- Resolution: Invalid Working as intended. The testcase ends up sending two batches, the first one being rejected and the second accepted. > First records in batch rejected but others accepted when rolling log > > > Key: KAFKA-5390 > URL: https://issues.apache.org/jira/browse/KAFKA-5390 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.2.1 > Reporter: Magnus Reftel > Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz > > > When sending a sequence of records in a batch right when the broker needs to > roll a new segment, it's possible for the first few records to fail, while > other records in the batch are accepted. If records have dependencies on > earlier records, e.g. in the case of a sequence of events in an event-sourced > system, then a producer cannot use the batching functionality, since it then > risks consumers receiving a record without first receiving the records it > depends on. > See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[
https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040449#comment-16040449
]
Magnus Reftel commented on KAFKA-5390:
--
Our records are typically small, but potentially huge. Doing manual batching
implicitly by ensuring that {{batch.size}} is greater than a fixed number of
worst-case records means tuning for the worst case instead of the expected
case. Also, the calculation still depends on protocol framing etc., which feels
wrong to assume things about outside of the client library. Setting
{{linger.ms}} to an arbitrary, large value in order to avoid batches being sent
too early also seems brittle - would adding a third way of deciding when to
send a batch in addition to the linger time and buffer size be regarded as an
API-breaking change?
I'd much rather do batching explicitly, e.g. by a sequence of calls to a
{{boolean addBatched(ProducerRecord, Callback}} method which indicates whether
there was buffer space for the records, followed by a {{void sendBatch()}}.
Would a patch adding that functionality be of interest?
> First records in batch rejected but others accepted when rolling log
>
>
> Key: KAFKA-5390
> URL: https://issues.apache.org/jira/browse/KAFKA-5390
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: Magnus Reftel
> Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz
>
>
> When sending a sequence of records in a batch right when the broker needs to
> roll a new segment, it's possible for the first few records to fail, while
> other records in the batch are accepted. If records have dependencies on
> earlier records, e.g. in the case of a sequence of events in an event-sourced
> system, then a producer cannot use the batching functionality, since it then
> risks consumers receiving a record without first receiving the records it
> depends on.
> See attached testcase (kafka-sequence.tar.gz).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[ https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038950#comment-16038950 ] Magnus Reftel commented on KAFKA-5390: -- Yes and no =) It's good to know that batches either succeed or fail as a whole, but in order to use batching in a producer, I'd have to have some way of knowing whether a new batch has been started. I could try to duplicate the code in the producer to see if there's enough space in the current batch for one more message, but that seems very brittle to me, since it depends both on the details of the serialization of messages and the linger time. Could the producer API be extended with a send method that fails if it would require a new batch, or something along those lines, so that producers know when to wait for ACK before continuing? > First records in batch rejected but others accepted when rolling log > > > Key: KAFKA-5390 > URL: https://issues.apache.org/jira/browse/KAFKA-5390 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.2.1 >Reporter: Magnus Reftel > Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz > > > When sending a sequence of records in a batch right when the broker needs to > roll a new segment, it's possible for the first few records to fail, while > other records in the batch are accepted. If records have dependencies on > earlier records, e.g. in the case of a sequence of events in an event-sourced > system, then a producer cannot use the batching functionality, since it then > risks consumers receiving a record without first receiving the records it > depends on. > See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[ https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038879#comment-16038879 ] Magnus Reftel edited comment on KAFKA-5390 at 6/6/17 1:28 PM: -- No, the server does not reject the batch - it rejects _some_ of the records in the batch, but not all. Further, the records of the batch that are accepted are not the first ones, so consumers would first see some records of earlier batches, then a gap where rejected records of the partially failing batch were, and then more records from the end of the batch. The producer can detect that this happens from seeing that there are some failures followed by some successful sends, but it's too late for it to do anything about it. was (Author: magnus.reftel): No, the server does not reject the batch - it rejects /some/ of the records, but not all. Further, the records of the batch that are accepted are not the first ones, so consumers would first see some records of earlier batches, then a gap where rejected records of the partially failing batch were, and then more records from the end of the batch. The producer can detect that this happens from seeing that there are some failures followed by some successful sends, but it's too late for it to do anything about it. > First records in batch rejected but others accepted when rolling log > > > Key: KAFKA-5390 > URL: https://issues.apache.org/jira/browse/KAFKA-5390 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.2.1 >Reporter: Magnus Reftel > Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz > > > When sending a sequence of records in a batch right when the broker needs to > roll a new segment, it's possible for the first few records to fail, while > other records in the batch are accepted. If records have dependencies on > earlier records, e.g. in the case of a sequence of events in an event-sourced > system, then a producer cannot use the batching functionality, since it then > risks consumers receiving a record without first receiving the records it > depends on. > See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[ https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038879#comment-16038879 ] Magnus Reftel commented on KAFKA-5390: -- No, the server does not reject the batch - it rejects /some/ of the records, but not all. Further, the records of the batch that are accepted are not the first ones, so consumers would first see some records of earlier batches, then a gap where rejected records of the partially failing batch were, and then more records from the end of the batch. The producer can detect that this happens from seeing that there are some failures followed by some successful sends, but it's too late for it to do anything about it. > First records in batch rejected but others accepted when rolling log > > > Key: KAFKA-5390 > URL: https://issues.apache.org/jira/browse/KAFKA-5390 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.2.1 >Reporter: Magnus Reftel > Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz > > > When sending a sequence of records in a batch right when the broker needs to > roll a new segment, it's possible for the first few records to fail, while > other records in the batch are accepted. If records have dependencies on > earlier records, e.g. in the case of a sequence of events in an event-sourced > system, then a producer cannot use the batching functionality, since it then > risks consumers receiving a record without first receiving the records it > depends on. > See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[ https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Reftel updated KAFKA-5390: - Attachment: kafka-sequence-2.tar.gz Updated to print results from the send callback. > First records in batch rejected but others accepted when rolling log > > > Key: KAFKA-5390 > URL: https://issues.apache.org/jira/browse/KAFKA-5390 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.2.1 > Reporter: Magnus Reftel > Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz > > > When sending a sequence of records in a batch right when the broker needs to > roll a new segment, it's possible for the first few records to fail, while > other records in the batch are accepted. If records have dependencies on > earlier records, e.g. in the case of a sequence of events in an event-sourced > system, then a producer cannot use the batching functionality, since it then > risks consumers receiving a record without first receiving the records it > depends on. > See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
[
https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038832#comment-16038832
]
Magnus Reftel commented on KAFKA-5390:
--
Thanks for the quick reply! I tried switching
{{records.forEach(producer::send);}} to
{code}
records.forEach(r -> producer.send(r, (m, e) -> {
System.out.printf("Send result for %s: %s\n", r.key(), m == null ?
"null" : "\"" + m + "\"");
if (e != null) {
e.printStackTrace();
}
}));
{code}
This produces the following output:
{noformat}
Send result for 0: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 1: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 2: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 3: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 4: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 5: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 6: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 7: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request
included message batch larger than the configured segment size on the server.
Send result for 8: "t-0@0"
{noformat}
The testcase still fails for the same reason as before.
> First records in batch rejected but others accepted when rolling log
>
>
> Key: KAFKA-5390
> URL: https://issues.apache.org/jira/browse/KAFKA-5390
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: Magnus Reftel
> Attachments: kafka-sequence.tar.gz
>
>
> When sending a sequence of records in a batch right when the broker needs to
> roll a new segment, it's possible for the first few records to fail, while
> other records in the batch are accepted. If records have dependencies on
> earlier records, e.g. in the case of a sequence of events in an event-sourced
> system, then a producer cannot use the batching functionality, since it then
> risks consumers receiving a record without first receiving the records it
> depends on.
> See attached testcase (kafka-sequence.tar.gz).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KAFKA-5390) First records in batch rejected but others accepted when rolling log
Magnus Reftel created KAFKA-5390: Summary: First records in batch rejected but others accepted when rolling log Key: KAFKA-5390 URL: https://issues.apache.org/jira/browse/KAFKA-5390 Project: Kafka Issue Type: Bug Affects Versions: 0.10.2.1 Reporter: Magnus Reftel Attachments: kafka-sequence.tar.gz When sending a sequence of records in a batch right when the broker needs to roll a new segment, it's possible for the first few records to fail, while other records in the batch are accepted. If records have dependencies on earlier records, e.g. in the case of a sequence of events in an event-sourced system, then a producer cannot use the batching functionality, since it then risks consumers receiving a record without first receiving the records it depends on. See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KAFKA-2495) Protocol guide only partially updated after ConsumerMetadata* addition?
[ https://issues.apache.org/jira/browse/KAFKA-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Reftel resolved KAFKA-2495. -- Resolution: Fixed > Protocol guide only partially updated after ConsumerMetadata* addition? > --- > > Key: KAFKA-2495 > URL: https://issues.apache.org/jira/browse/KAFKA-2495 > Project: Kafka > Issue Type: Improvement > Components: website > Reporter: Magnus Reftel > > It appears that the protocol guide has only been partially updated after the > ConsumerMetadataRequest and ConsumerMetadataResponse was added. In > particular, the response for a TopicMetadataRequest is called > "MetadataRequest" instead of "TopicMetadataRequest", and ResponseMessage > lists only MetadataResponse as an alternative, but should probably list > "TopicMetadataResponse" and "ConsumerMetadataResponse" instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KAFKA-2494) Document ReplicaId in OffsetRequest in the protocol guide
[ https://issues.apache.org/jira/browse/KAFKA-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Reftel resolved KAFKA-2494. -- Resolution: Fixed The field is documented, at least since 14ffd37c0d2a5fa1d05bda14110f0057238bbdee > Document ReplicaId in OffsetRequest in the protocol guide > - > > Key: KAFKA-2494 > URL: https://issues.apache.org/jira/browse/KAFKA-2494 > Project: Kafka > Issue Type: Improvement > Components: website > Reporter: Magnus Reftel > > The documentation for OffsetRequest in the protocol guide lists ReplicaId as > one of the fields, but does not say what the field is for. It appears that > it's similar to FetchRequest, where it's set to -1 by clients. It would be > nice if that was documented. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-5359) Exceptions from RequestFuture lack parts of the stack trace
Magnus Reftel created KAFKA-5359:
Summary: Exceptions from RequestFuture lack parts of the stack
trace
Key: KAFKA-5359
URL: https://issues.apache.org/jira/browse/KAFKA-5359
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: Magnus Reftel
Priority: Minor
When an exception occurs within a task that reports its result using a
RequestFuture, that exception is stored in a field on the RequestFuture using
the {{raise}} method. In many places in the code where such futures are
completed, that exception is then thrown directly using {{throw
future.exception();}} (see e.g.
[Fetcher.getTopicMetadata|https://github.com/apache/kafka/blob/aebba89a2b9b5ea6a7cab2599555232ef3fe21ad/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L316]).
This means that the exception that ends up in client code only has stack traces
related to the original exception, but nothing leading up to the completion of
the future. The client therefore gets no indication of what was going on in the
client code - only that it somehow ended up in the Kafka libraries, and that a
task failed at some point.
One solution to this is to use the exceptions from the future as causes for
chained exceptions, so that the client gets a stack trace that shows what the
client was doing, in addition to getting the stack traces for the exception in
the task.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] kafka pull request #3167: MINOR: Remove unused configuration setting
GitHub user reftel opened a pull request: https://github.com/apache/kafka/pull/3167 MINOR: Remove unused configuration setting The configuration setting zookeeper.sync.time.ms was not used for anything. Remove it to reduce confusion. This changes rebalance.backoff.ms from defaulting to the value of zookeeper.sync.time.ms to default to 2000 (zookeeper.sync.time.ms's default). This means that if someone has not set rebalance.backoff.ms but has changed zookeeper.sync.time.ms from its default, then rebalance.backoff.ms will now instead be 2000ms. I'd be surprised if there were users who do this intentionally. You can merge this pull request into a Git repository by running: $ git pull https://github.com/reftel/kafka feature/no_zk_sync_time Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3167 commit e7672bcd61193bc94a34ebc5368bcf8f510c1fbc Author: Magnus Reftel Date: 2017-05-30T12:59:49Z MINOR: Remove unused configuration setting The configuration setting zookeeper.sync.time.ms was not used for anything. Remove it to reduce confusion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2335: TRIVIAL: Fix spelling of log message
GitHub user reftel opened a pull request: https://github.com/apache/kafka/pull/2335 TRIVIAL: Fix spelling of log message You can merge this pull request into a Git repository by running: $ git pull https://github.com/reftel/kafka feature/triggering Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2335.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2335 commit 3700a0feeb14518732e2fc5eb60c447f8262116a Author: Magnus Reftel Date: 2017-01-09T12:39:27Z TRIVIAL: Fix spelling of log message --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration
[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693153#comment-15693153 ] Magnus Reftel commented on KAFKA-4406: -- I'd be glad to write one if there's interest. > Add support for custom Java Security Providers in configuration > --- > > Key: KAFKA-4406 > URL: https://issues.apache.org/jira/browse/KAFKA-4406 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.10.0.1 >Reporter: Magnus Reftel >Priority: Minor > > Currently, the only way to add a custom security provider is though adding a > -Djava.security.properties= option to the command line, e.g. though > KAFKA_OPTS. It would be more convenient if this could be done though the > config file, like all the other SSL related options. > I propose adding a new configuration option, ssl.provider.classes, which > holds a list of names of security provider classes that will be loaded, > instantiated, and added before creating SSL contexts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request #2151: Feature/authorizer name reference
GitHub user reftel opened a pull request: https://github.com/apache/kafka/pull/2151 Feature/authorizer name reference You can merge this pull request into a Git repository by running: $ git pull https://github.com/reftel/kafka feature/authorizer_name_reference Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2151 commit 30bb6b6725d48386dad51e66df4fd4c5f48c5066 Author: Magnus Reftel Date: 2016-11-19T13:37:24Z MINOR: editorial changes to the Authorizer javadoc commit bb0d9bf49ce2d636ce8256ca5b54e150ed3a12b7 Author: Magnus Reftel Date: 2016-11-19T13:37:24Z MINOR: fix javadoc reference to authorizer config Use the correct config option name, "authorizer.class.name". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration
[
https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670468#comment-15670468
]
Magnus Reftel commented on KAFKA-4406:
--
Right, so `ssl.` is out. How about `security.providers`?
For SASL, another option would be that if the user specified a preference of
provider, then get the factory class manually via
`Class.forName(Security.getProvider(configuredName).get("SaslClientFactory." +
configuredMechanism))`. That way, we'd be able to use provider names in all
places where we use the JCA provider mechanism.
How about the use of MessageDigest that you pointed out (only used by
SkimpyOffsetMap.scala it seems)? Should there be some way of selecting provider
there as well, for consistency?
> Add support for custom Java Security Providers in configuration
> ---
>
> Key: KAFKA-4406
> URL: https://issues.apache.org/jira/browse/KAFKA-4406
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Affects Versions: 0.10.0.1
>Reporter: Magnus Reftel
>Priority: Minor
>
> Currently, the only way to add a custom security provider is though adding a
> -Djava.security.properties= option to the command line, e.g. though
> KAFKA_OPTS. It would be more convenient if this could be done though the
> config file, like all the other SSL related options.
> I propose adding a new configuration option, ssl.provider.classes, which
> holds a list of names of security provider classes that will be loaded,
> instantiated, and added before creating SSL contexts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration
[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670080#comment-15670080 ] Magnus Reftel commented on KAFKA-4406: -- The PR adds it for both broker and clients, so both can be configured using the same property, if desired. The name of the property could definitely be better, though, as it's not tied to ssl.provider (which is used as provider name for the SSL context, and the SSL context only, whereas the provider classes are relevant to all algorithm lookups). All other settings related to security start with `ssl.`, though, so it should probably be in that part of the tree (and I find no usages of JCA provider mechanism outside of the SSL code). Any suggestions? I don't quite get what the use of replacing an existing provider would be. If overriding the implementation of an existing algorithm, then specifying the provider name (like one can do for SSL contexts using the `ssl.provider` setting - I guess this would be useful also in the other places where one can specify algorithm names) is the standard way of doing it. If adding a new algorithm, then using the name of the new one as e.g. `ssl.keymanager.algorithm` would suffice. What am I missing? > Add support for custom Java Security Providers in configuration > --- > > Key: KAFKA-4406 > URL: https://issues.apache.org/jira/browse/KAFKA-4406 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.10.0.1 >Reporter: Magnus Reftel >Priority: Minor > > Currently, the only way to add a custom security provider is though adding a > -Djava.security.properties= option to the command line, e.g. though > KAFKA_OPTS. It would be more convenient if this could be done though the > config file, like all the other SSL related options. > I propose adding a new configuration option, ssl.provider.classes, which > holds a list of names of security provider classes that will be loaded, > instantiated, and added before creating SSL contexts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration
[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667481#comment-15667481 ] Magnus Reftel commented on KAFKA-4406: -- For client applications, I agree that it is better to just call `Security.addProvider` directly, which is what we do. The purpose of this change is to allow configuration files to add security providers to the Kafka brokers. The alternative there is, as far as I can see, only to add a `-Djava.security.properties`, guessing at a free provider index, and hoping that it stays unused over time. That doesn't seem like a reliable solution to me. > Add support for custom Java Security Providers in configuration > --- > > Key: KAFKA-4406 > URL: https://issues.apache.org/jira/browse/KAFKA-4406 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.10.0.1 >Reporter: Magnus Reftel >Priority: Minor > > Currently, the only way to add a custom security provider is though adding a > -Djava.security.properties= option to the command line, e.g. though > KAFKA_OPTS. It would be more convenient if this could be done though the > config file, like all the other SSL related options. > I propose adding a new configuration option, ssl.provider.classes, which > holds a list of names of security provider classes that will be loaded, > instantiated, and added before creating SSL contexts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-4406) Add support for custom Java Security Providers in configuration
[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Reftel updated KAFKA-4406: - Status: Patch Available (was: Open) https://github.com/apache/kafka/pull/2134 > Add support for custom Java Security Providers in configuration > --- > > Key: KAFKA-4406 > URL: https://issues.apache.org/jira/browse/KAFKA-4406 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.10.0.1 > Reporter: Magnus Reftel >Priority: Minor > > Currently, the only way to add a custom security provider is though adding a > -Djava.security.properties= option to the command line, e.g. though > KAFKA_OPTS. It would be more convenient if this could be done though the > config file, like all the other SSL related options. > I propose adding a new configuration option, ssl.provider.classes, which > holds a list of names of security provider classes that will be loaded, > instantiated, and added before creating SSL contexts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (KAFKA-4406) Add support for custom Java Security Providers in configuration
[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Reftel updated KAFKA-4406: - Comment: was deleted (was: https://github.com/apache/kafka/pull/2134) > Add support for custom Java Security Providers in configuration > --- > > Key: KAFKA-4406 > URL: https://issues.apache.org/jira/browse/KAFKA-4406 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.10.0.1 > Reporter: Magnus Reftel >Priority: Minor > > Currently, the only way to add a custom security provider is though adding a > -Djava.security.properties= option to the command line, e.g. though > KAFKA_OPTS. It would be more convenient if this could be done though the > config file, like all the other SSL related options. > I propose adding a new configuration option, ssl.provider.classes, which > holds a list of names of security provider classes that will be loaded, > instantiated, and added before creating SSL contexts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration
[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15666899#comment-15666899 ] Magnus Reftel commented on KAFKA-4406: -- https://github.com/apache/kafka/pull/2134 > Add support for custom Java Security Providers in configuration > --- > > Key: KAFKA-4406 > URL: https://issues.apache.org/jira/browse/KAFKA-4406 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.10.0.1 > Reporter: Magnus Reftel >Priority: Minor > > Currently, the only way to add a custom security provider is though adding a > -Djava.security.properties= option to the command line, e.g. though > KAFKA_OPTS. It would be more convenient if this could be done though the > config file, like all the other SSL related options. > I propose adding a new configuration option, ssl.provider.classes, which > holds a list of names of security provider classes that will be loaded, > instantiated, and added before creating SSL contexts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request #2134: config: add ssl.provider.classes config option
GitHub user reftel opened a pull request: https://github.com/apache/kafka/pull/2134 config: add ssl.provider.classes config option You can merge this pull request into a Git repository by running: $ git pull https://github.com/reftel/kafka feature/security_providers_config Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2134.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2134 commit 11bd07fd7a27ebc8e48b7c895e3a1ce987131041 Author: Magnus Reftel Date: 2016-11-14T12:45:18Z config: add ssl.provider.classes config option --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (KAFKA-4406) Add support for custom Java Security Providers in configuration
Magnus Reftel created KAFKA-4406: Summary: Add support for custom Java Security Providers in configuration Key: KAFKA-4406 URL: https://issues.apache.org/jira/browse/KAFKA-4406 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.10.0.1 Reporter: Magnus Reftel Priority: Minor Currently, the only way to add a custom security provider is though adding a -Djava.security.properties= option to the command line, e.g. though KAFKA_OPTS. It would be more convenient if this could be done though the config file, like all the other SSL related options. I propose adding a new configuration option, ssl.provider.classes, which holds a list of names of security provider classes that will be loaded, instantiated, and added before creating SSL contexts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request #1901: MINOR: Fix Javadoc for KafkaConsumer.poll
GitHub user reftel opened a pull request: https://github.com/apache/kafka/pull/1901 MINOR: Fix Javadoc for KafkaConsumer.poll You can merge this pull request into a Git repository by running: $ git pull https://github.com/reftel/kafka feature/poll_javadoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1901.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1901 commit 0be3abbda9a97c6f3d135226d3e38755aed7f283 Author: Magnus Reftel Date: 2016-09-23T07:02:59Z MINOR: Fix Javadoc for KafkaConsumer.poll --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (KAFKA-2494) Document ReplicaId in OffsetRequest in the protocol guide
[ https://issues.apache.org/jira/browse/KAFKA-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Reftel updated KAFKA-2494: - Summary: Document ReplicaId in OffsetRequest in the protocol guide (was: Documetn ReplicaId in OffsetRequest in the protocol guide) > Document ReplicaId in OffsetRequest in the protocol guide > - > > Key: KAFKA-2494 > URL: https://issues.apache.org/jira/browse/KAFKA-2494 > Project: Kafka > Issue Type: Improvement > Components: website > Reporter: Magnus Reftel > > The documentation for OffsetRequest in the protocol guide lists ReplicaId as > one of the fields, but does not say what the field is for. It appears that > it's similar to FetchRequest, where it's set to -1 by clients. It would be > nice if that was documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2495) Protocol guide only partially updated after ConsumerMetadata* addition?
Magnus Reftel created KAFKA-2495: Summary: Protocol guide only partially updated after ConsumerMetadata* addition? Key: KAFKA-2495 URL: https://issues.apache.org/jira/browse/KAFKA-2495 Project: Kafka Issue Type: Improvement Components: website Reporter: Magnus Reftel It appears that the protocol guide has only been partially updated after the ConsumerMetadataRequest and ConsumerMetadataResponse was added. In particular, the response for a TopicMetadataRequest is called "MetadataRequest" instead of "TopicMetadataRequest", and ResponseMessage lists only MetadataResponse as an alternative, but should probably list "TopicMetadataResponse" and "ConsumerMetadataResponse" instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2494) Documetn ReplicaId in OffsetRequest in the protocol guide
Magnus Reftel created KAFKA-2494: Summary: Documetn ReplicaId in OffsetRequest in the protocol guide Key: KAFKA-2494 URL: https://issues.apache.org/jira/browse/KAFKA-2494 Project: Kafka Issue Type: Improvement Components: website Reporter: Magnus Reftel The documentation for OffsetRequest in the protocol guide lists ReplicaId as one of the fields, but does not say what the field is for. It appears that it's similar to FetchRequest, where it's set to -1 by clients. It would be nice if that was documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
