[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013975#comment-16013975 ] Yaniv Kunda commented on KAFKA-1995: You can check out Confluent's JMS Client for Kafka: http://docs.confluent.io/current/clients/kafka-jms-client/docs/index.html > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.9.0.0 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15315665#comment-15315665 ] Ewen Cheslack-Postava commented on KAFKA-1995: -- [~johnlon] JMS supports acknowledged messages. Message IDs and correlation IDs could potentially be used as offsets. Given some other guarantees, offsets don't necessarily need to be comparable and the source system doesn't need to be able to seek to arbitrary offsets. In the case of JMS, I think you could do something like use the message ID as the offset and if a Connect offset commit occurs but you don't have a chance to ack that block of messages, upon resuming you could consume messages, ignoring anything until you see the ID from the last commit, then start producing data. This minimizes the number of duplicates you might see in the destination system as long as you don't ack messages until Connect offsets have been committed. Alternatively, opt to ignore offsets and offset commit entirely and rely only on commitRecord (https://github.com/apache/kafka/blob/trunk/connect/api/src/main/java/org/apache/kafka/connect/source/SourceTask.java#L84) to trigger acks. Upon faults you can still see duplicates, but since JMS has fine-grained acking and effectively tracks offsets for you, you may not even need Connect's offset tracking support. That support is included for systems that have no other place to store offsets (e.g. if you're reading of a database's commit log file), but it isn't required that every connector utilize it. Strictly speaking you can also create a connector that doesn't use these features, you just won't have any guarantees about data delivery in the case of faults. > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.9.0.0 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311268#comment-15311268 ] John Lonergan commented on KAFKA-1995: -- Is it problematic that JMS reads (rather the commits) are destructive. Also there is no obvious JMS feature that could be used for the source offset. > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.9.0.0 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169426#comment-15169426 ] James Cheng commented on KAFKA-1995: Now that Kafka Connect has been released, a good way to do this would be by writing a JMS Kafka Connector. Someone mentioned MQTT. I don't know if that's the same thing as JMS, but someone released a MQTT Kafka Connector. See http://www.confluent.io/developers/connectors > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.9.0.0 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168649#comment-15168649 ] Christian Ferrari commented on KAFKA-1995: -- A JMS bridge would be very helpful to support this pattern: [MQTT client] ---(publish)---> [MQTT/JMS broker] ---(Kafka JMS bridge)--> [Kafka] --> ... > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.9.0.0 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349493#comment-14349493 ] Rekha Joshi commented on KAFKA-1995: Great.Thanks for your reply, [~ewencp] [~joestein] ! > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.8.3 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348951#comment-14348951 ] Joe Stein commented on KAFKA-1995: -- << This sounds like a good idea, but I think it doesn't belong in Kafka itself Agreed on both A JMS client would be great https://cwiki.apache.org/confluence/display/KAFKA/Clients for sure. > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.8.3 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348342#comment-14348342 ] Ewen Cheslack-Postava commented on KAFKA-1995: -- This sounds like a good idea, but I think it doesn't belong in Kafka itself. This sounds a lot like a bunch of other connectors that allow you to pipe data through other systems: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem I don't know JMS well enough to give suggestions about the implementation, but it seems like the translation of JMS provider -> Kafka, JMS producer -> Kafka producer and JMS consumer -> Kafka consumer should be a pretty minimal, simple layer. You might have better luck getting design feedback on the mailing lists. > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.8.3 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1995) JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but hit Kafka)
[ https://issues.apache.org/jira/browse/KAFKA-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342819#comment-14342819 ] Rekha Joshi commented on KAFKA-1995: [~jkreps] [~nehanarkhede] [~junrao] [~joestein] it would be great to know your thoughts on it.thanks. > JMS to Kafka: Inbuilt JMSAdaptor/JMSProxy/JMSBridge (Client can speak JMS but > hit Kafka) > > > Key: KAFKA-1995 > URL: https://issues.apache.org/jira/browse/KAFKA-1995 > Project: Kafka > Issue Type: Wish > Components: core >Affects Versions: 0.8.3 >Reporter: Rekha Joshi > > Kafka is a great alternative to JMS, providing high performance, throughput > as scalable, distributed pub sub/commit log service. > However there always exist traditional systems running on JMS. > Rather than rewriting, it would be great if we just had an inbuilt > JMSAdaptor/JMSProxy/JMSBridge by which client can speak JMS but hit Kafka > behind-the-scene. > Something like Chukwa's o.a.h.chukwa.datacollection.adaptor.jms.JMSAdaptor, > which receives msg off JMS queue and transforms to a Chukwa chunk? > I have come across folks talking of this need in past as well.Is it > considered and/or part of the roadmap? > http://grokbase.com/t/kafka/users/131cst8xpv/stomp-binding-for-kafka > http://grokbase.com/t/kafka/users/148dm4247q/consuming-messages-from-kafka-and-pushing-on-to-a-jms-queue > http://grokbase.com/t/kafka/users/143hjepbn2/request-kafka-zookeeper-jms-details > Looking for inputs on correct way to approach this so to retain all good > features of Kafka while still not rewriting entire application.Possible? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
