[jira] [Updated] (KAFKA-7590) GETTING HUGE MESSAGE STRUCTURE THROUGH JMS CONNECTOR
[
https://issues.apache.org/jira/browse/KAFKA-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chenchu Lakshman kumar updated KAFKA-7590:
--
Description:
Message
{"schema":{"type":"struct","fields":[
{"type":"string","optional":false,"doc":"This field stores the value of
`Message.getJMSMessageID()
`_.","field":"messageID"}
,
{"type":"string","optional":false,"doc":"This field stores the type of message
that was received. This corresponds to the subinterfaces of `Message
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html]>`_. `BytesMessage
<[http://docs.oracle.com/javaee/6/api/javax/jms/BytesMessage.html]>`_ =
`bytes`, `MapMessage
<[http://docs.oracle.com/javaee/6/api/javax/jms/MapMessage.html]>`_ = `map`,
`ObjectMessage
<[http://docs.oracle.com/javaee/6/api/javax/jms/ObjectMessage.html]>`_ =
`object`, `StreamMessage
<[http://docs.oracle.com/javaee/6/api/javax/jms/StreamMessage.html]>`_ =
`stream` and `TextMessage
<[http://docs.oracle.com/javaee/6/api/javax/jms/TextMessage.html]>`_ = `text`.
The corresponding field will be populated with the values from the respective
Message subinterface.","field":"messageType"}
,
{"type":"int64","optional":false,"doc":"Data from the `getJMSTimestamp()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSTimestamp(])>`_
method.","field":"timestamp"}
,
{"type":"int32","optional":false,"doc":"This field stores the value of
`Message.getJMSDeliveryMode()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSDeliveryMode(])>`_.","field":"deliveryMode"}
,
{"type":"string","optional":true,"doc":"This field stores the value of
`Message.getJMSCorrelationID()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSCorrelationID(])>`_.","field":"correlationID"}
,{"type":"struct","fields":[
{"type":"string","optional":false,"doc":"The type of JMS Destination, and
either ``queue`` or ``topic``.","field":"destinationType"}
,
{"type":"string","optional":false,"doc":"The name of the destination. This will
be the value of `Queue.getQueueName()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html#getQueueName(])>`_
or `Topic.getTopicName()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html#getTopicName(])>`_.","field":"name"}
],"optional":true,"name":"io.confluent.connect.jms.Destination","doc":"This
schema is used to represent a JMS Destination, and is either `queue
<[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html]>`_ or `topic
<[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html]>`_.","field":"replyTo"},{"type":"struct","fields":[
{"type":"string","optional":false,"doc":"The type of JMS Destination, and
either ``queue`` or ``topic``.","field":"destinationType"}
,
{"type":"string","optional":false,"doc":"The name of the destination. This will
be the value of `Queue.getQueueName()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html#getQueueName(])>`_
or `Topic.getTopicName()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html#getTopicName(])>`_.","field":"name"}
],"optional":true,"name":"io.confluent.connect.jms.Destination","doc":"This
schema is used to represent a JMS Destination, and is either `queue
<[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html]>`_ or `topic
<[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html]>`_.","field":"destination"},
{"type":"boolean","optional":false,"doc":"This field stores the value of
`Message.getJMSRedelivered()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSRedelivered(])>`_.","field":"redelivered"}
,
{"type":"string","optional":true,"doc":"This field stores the value of
`Message.getJMSType()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSType(])>`_.","field":"type"}
,
{"type":"int64","optional":false,"doc":"This field stores the value of
`Message.getJMSExpiration()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSExpiration(])>`_.","field":"expiration"}
,
{"type":"int32","optional":false,"doc":"This field stores the value of
`Message.getJMSPriority()
<[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSPriority(])>`_.","field":"priority"}
,{"type":"map","keys":
{"type":"string","optional":false}
,"values":{"type":"struct","fields":[
{"type":"string","optional":false,"doc":"The java type of the property on the
Message. One of ``boolean``, ``byte``, ``short``, ``integer``, ``long``,
``float``, ``double``, or ``string``.","field":"propertyType"}
,\{"type":"boolean","optional":true,"doc":"The value stored as a boolean. Null
unless ``propertyType`` is set to
``boolean``.","field":"boolean"},\{"type":"int8","optional":true,"doc":"The
value stored as a byte. Null unless
[jira] [Resolved] (KAFKA-7590) GETTING HUGE MESSAGE STRUCTURE THROUGH JMS CONNECTOR
[
https://issues.apache.org/jira/browse/KAFKA-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chenchu Lakshman kumar resolved KAFKA-7590.
---
Resolution: Fixed
> GETTING HUGE MESSAGE STRUCTURE THROUGH JMS CONNECTOR
>
>
> Key: KAFKA-7590
> URL: https://issues.apache.org/jira/browse/KAFKA-7590
> Project: Kafka
> Issue Type: Test
> Components: config, KafkaConnect
>Affects Versions: 2.0.0
>Reporter: Chenchu Lakshman kumar
>Priority: Major
>
> Message
>
> {"schema":{"type":"struct","fields":[
> {"type":"string","optional":false,"doc":"This field stores the value of
> `Message.getJMSMessageID()
> `_.","field":"messageID"}
> ,
> {"type":"string","optional":false,"doc":"This field stores the type of
> message that was received. This corresponds to the subinterfaces of `Message
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html]>`_.
> `BytesMessage
> <[http://docs.oracle.com/javaee/6/api/javax/jms/BytesMessage.html]>`_ =
> `bytes`, `MapMessage
> <[http://docs.oracle.com/javaee/6/api/javax/jms/MapMessage.html]>`_ = `map`,
> `ObjectMessage
> <[http://docs.oracle.com/javaee/6/api/javax/jms/ObjectMessage.html]>`_ =
> `object`, `StreamMessage
> <[http://docs.oracle.com/javaee/6/api/javax/jms/StreamMessage.html]>`_ =
> `stream` and `TextMessage
> <[http://docs.oracle.com/javaee/6/api/javax/jms/TextMessage.html]>`_ =
> `text`. The corresponding field will be populated with the values from the
> respective Message subinterface.","field":"messageType"}
> ,
> {"type":"int64","optional":false,"doc":"Data from the `getJMSTimestamp()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSTimestamp(])>`_
> method.","field":"timestamp"}
> ,
> {"type":"int32","optional":false,"doc":"This field stores the value of
> `Message.getJMSDeliveryMode()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSDeliveryMode(])>`_.","field":"deliveryMode"}
> ,
> {"type":"string","optional":true,"doc":"This field stores the value of
> `Message.getJMSCorrelationID()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSCorrelationID(])>`_.","field":"correlationID"}
> ,{"type":"struct","fields":[
> {"type":"string","optional":false,"doc":"The type of JMS Destination, and
> either ``queue`` or ``topic``.","field":"destinationType"}
> ,
> {"type":"string","optional":false,"doc":"The name of the destination. This
> will be the value of `Queue.getQueueName()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html#getQueueName(])>`_
> or `Topic.getTopicName()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html#getTopicName(])>`_.","field":"name"}
> ],"optional":true,"name":"io.confluent.connect.jms.Destination","doc":"This
> schema is used to represent a JMS Destination, and is either `queue
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html]>`_ or `topic
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html]>`_.","field":"replyTo"},{"type":"struct","fields":[
> {"type":"string","optional":false,"doc":"The type of JMS Destination, and
> either ``queue`` or ``topic``.","field":"destinationType"}
> ,
> {"type":"string","optional":false,"doc":"The name of the destination. This
> will be the value of `Queue.getQueueName()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html#getQueueName(])>`_
> or `Topic.getTopicName()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html#getTopicName(])>`_.","field":"name"}
> ],"optional":true,"name":"io.confluent.connect.jms.Destination","doc":"This
> schema is used to represent a JMS Destination, and is either `queue
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html]>`_ or `topic
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Topic.html]>`_.","field":"destination"},
> {"type":"boolean","optional":false,"doc":"This field stores the value of
> `Message.getJMSRedelivered()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSRedelivered(])>`_.","field":"redelivered"}
> ,
> {"type":"string","optional":true,"doc":"This field stores the value of
> `Message.getJMSType()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSType(])>`_.","field":"type"}
> ,
> {"type":"int64","optional":false,"doc":"This field stores the value of
> `Message.getJMSExpiration()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSExpiration(])>`_.","field":"expiration"}
> ,
> {"type":"int32","optional":false,"doc":"This field stores the value of
> `Message.getJMSPriority()
> <[http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSPriority(])>`_.","field":"priority"}
>
[jira] [Created] (KAFKA-7616) MockConsumer can return ConsumerRecords objects with a non-empty map but no records
Stig Rohde Døssing created KAFKA-7616: - Summary: MockConsumer can return ConsumerRecords objects with a non-empty map but no records Key: KAFKA-7616 URL: https://issues.apache.org/jira/browse/KAFKA-7616 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 2.0.1 Reporter: Stig Rohde Døssing Assignee: Stig Rohde Døssing The ConsumerRecords returned from MockConsumer.poll can return false for isEmpty while not containing any records. This behavior is because MockConsumer.poll eagerly adds entries to the returned Map>, based on which partitions have been added. If no records are returned for a partition, e.g. because the position was too far ahead, the entry for that partition will still be there. The MockConsumer should lazily add entries to the map as they are needed, since it is more in line with how the real consumer behaves. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7615) Support different topic name in source and destination server in Mirrormaker
[ https://issues.apache.org/jira/browse/KAFKA-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adeeti Kaushal updated KAFKA-7615: -- Description: Currently mirrormaker only supports same topic name in source and destination broker. Support for different topic names in source and destination brokers is needed. source broker : topic name -> topicA destination broker: topic name -> topicB MirrorData from topicA to topicB was: Currently mirrormaker only supports same topic name in source and destination broker. Support for different topic names in source and destination brokers is needed. source broker : topic name -> topicA destination broker: topic name -> topicA_new-- > Support different topic name in source and destination server in Mirrormaker > > > Key: KAFKA-7615 > URL: https://issues.apache.org/jira/browse/KAFKA-7615 > Project: Kafka > Issue Type: New Feature > Components: mirrormaker >Reporter: Adeeti Kaushal >Priority: Minor > > Currently mirrormaker only supports same topic name in source and destination > broker. Support for different topic names in source and destination brokers > is needed. > > source broker : topic name -> topicA > destination broker: topic name -> topicB > > MirrorData from topicA to topicB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7615) Support different topic name in source and destination server in Mirrormaker
Adeeti Kaushal created KAFKA-7615: - Summary: Support different topic name in source and destination server in Mirrormaker Key: KAFKA-7615 URL: https://issues.apache.org/jira/browse/KAFKA-7615 Project: Kafka Issue Type: New Feature Components: mirrormaker Reporter: Adeeti Kaushal Currently mirrormaker only supports same topic name in source and destination broker. Support for different topic names in source and destination brokers is needed. source broker : topic name -> topicA destination broker: topic name -> topicA_new-- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-6812) Async ConsoleProducer exits with 0 status even after data loss
[
https://issues.apache.org/jira/browse/KAFKA-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Kang reassigned KAFKA-6812:
-
Assignee: Kamal Kang (was: Stanislav Kozlovski)
> Async ConsoleProducer exits with 0 status even after data loss
> --
>
> Key: KAFKA-6812
> URL: https://issues.apache.org/jira/browse/KAFKA-6812
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 1.1.0
>Reporter: Andras Beni
>Assignee: Kamal Kang
>Priority: Minor
>
> When {{ConsoleProducer}} is run without {{--sync}} flag and one of the
> batches times out, {{ErrorLoggingCallback}} logs the error:
> {code:java}
> 18/04/21 04:23:01 WARN clients.NetworkClient: [Producer
> clientId=console-producer] Connection to node 10 could not be established.
> Broker may not be available.
> 18/04/21 04:23:02 ERROR internals.ErrorLoggingCallback: Error when sending
> message to topic my-topic with key: null, value: 8 bytes with error:
> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for
> my-topic-0: 1530 ms has passed since batch creation plus linger time{code}
> However, the tool exits with status code 0.
> In my opinion the tool should indicate in the exit status that there was
> data lost. Maybe it's reasonable to exit after the first error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7565) NPE in KafkaConsumer
[
https://issues.apache.org/jira/browse/KAFKA-7565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682850#comment-16682850
]
Alexey Vakhrenev commented on KAFKA-7565:
-
I'll try it next week and report back. Thank you!
> NPE in KafkaConsumer
>
>
> Key: KAFKA-7565
> URL: https://issues.apache.org/jira/browse/KAFKA-7565
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 1.1.1
>Reporter: Alexey Vakhrenev
>Priority: Critical
> Fix For: 2.2.0
>
>
> The stacktrace is
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:221)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:202)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:563)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:390)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:244)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1171)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
> {noformat}
> Couldn't find minimal reproducer, but it happens quite often in our system.
> We use {{pause()}} and {{wakeup()}} methods quite extensively, maybe it is
> somehow related.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7612) Fix javac warnings and enable warnings as errors
[ https://issues.apache.org/jira/browse/KAFKA-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682824#comment-16682824 ] ASF GitHub Bot commented on KAFKA-7612: --- ijuma opened a new pull request #5900: KAFKA-7612: Fix javac warnings and enable warnings as errors URL: https://github.com/apache/kafka/pull/5900 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix javac warnings and enable warnings as errors > > > Key: KAFKA-7612 > URL: https://issues.apache.org/jira/browse/KAFKA-7612 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > > The only way to keep warnings away is to treat them as errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
