I've used both MQ and Kafka so I'll give a brief overview of the
differences and capabilities.
Kafka is distributed system. It's designed to be scaled horizontally for
both fail-over and throughput. It uses a consensus algorithm similar to
RAFT to elect a leader both initially and in the event of a broker
failure when it needs to re-balance the cluster. MQ is a traditional
message broker where Kafka is more of a streaming platform that you can
think of a distributed log. The Kafka unit of scale is a partition which
is a file in the file system, distributed either randomly using keys or
round robin. MQ is push based with a local transaction log where Kafka
is pull based using commits to keep the offset into the log per
consumer. The benefit of this is that you can take a service down for
upgrade and then restart from the last commit point. Unlike MQ Kafka
keeps doesn't delete records when they are consumed so you can bring new
services online that start from the earliest position in the log. Data
retention is managed using policies such as time and size limits.
Kafka works in a sysplex as it boils down to a TCP based protocol and is
platform agnostic. In fact you could build a cluster with some brokers
on z/OS and some on x86 Linux if you wanted to. It runs well on z/OS,
although it was broken until last year as z/OS UNIX mmap was limited to
31-bit address spaces. IBM fixed the issue as Kafka is strategic for
analytics products (including ours). There is some doc
https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=configuring-setting-up-apache-kafka-z.
Where the two really start to differ is the eco-system. Kafka is much
more then just a broker. There is a stream API and other abstractions to
layer on top such as ksqlDB which a SQL interface to live data streams.
You write a query which can aggregate/group data from a stream based
upon a time window and then emit results to an output topic. For
example, you can stream SMF 110 CICS CMF records, aggregate them into a
5 minute hopping window with AVG, MIN, MAX functions etc and then
publish them to Elasticsearch, Splunk or a data lake.
https://docs.ksqldb.io/en/latest/concepts/time-and-windows-in-ksqldb-queries/
MQ is the clear winner for traditional mainframe transactions, Kafka is
the goto for real time streaming of events.
On 20/06/2022 9:53 pm, Colin Paice wrote:
On the MQ front, I >think< there are customers who are using QREP to do DB2
to DB2 using MQ as the transport, over 1000 KM and getting about 100MB+ of
data a second - with a few second or subsecond response time. So to the
end user it looks like a single system rather than replicated.
The limits are the rate at which you can log to disk - about 200 MB a
second last time I was involved, and the capacity of the network.
I dont know abou Kafta, if it supports sysplex, and what it's throughput is
in similar scenarios.
If you want more throughput you create another queue manager, and get more
network capacity.
Colin
On Mon, 20 Jun 2022 at 14:38, Gerhard Adam <gada...@charter.net> wrote:
Perhaps no one has :grokked" the difference is because either there isn't
one or because it is so poorly explained and discussed as to be
non-existent. This sounds like more marketing hype perpetrated by
individuals that know buzzwords and little else.
-----Original Message-----
From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On Behalf
Of David Crayford
Sent: Monday, June 20, 2022 6:12 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Modernize Mainframe Applications for Hybrid Cloud with IBM
and AWS
Thanks. I’ve seen something similar on the ACM
https://dl.acm.org/doi/pdf/10.1145/1476793.1476796
On 20 Jun 2022, at 8:51 pm, Joe Monk <joemon...@gmail.com> wrote:
https://ntrs.nasa.gov/citations/19690000381
Joe
On Mon, Jun 20, 2022 at 7:00 AM David Crayford <dcrayf...@gmail.com>
wrote:
And yet still nobody seems to have grokked the fundamental
differences between online systems and event-driven architectures.
This is obviously not the forum for discussions on contemporary
software architectures. It always deteriorates into a deluge of
boring and undiscerningposts about how it's nothing new and was
already done back in the day on a S360 with 4K ram and a paper-taper
reader held together with gaffer tape.
https://en.wikipedia.org/wiki/Event-driven_architecture
On 20/06/2022 7:40 pm, Seymour J Metz wrote:
Before MQ there were QTAM and TCAM.
<groan>
There have been mainframes running real time applications since the
1960s. Air traffic control. Airline reservations. Controlling traffic
lights (UNIVAC, not IBM.) These may not be the best examples, but
they were the first to come to mind, and used off the shelf mainframes.
--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
________________________________________
From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on
behalf of René Jansen [rene.vincent.jan...@gmail.com]
Sent: Monday, June 20, 2022 5:50 AM
To:IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Modernize Mainframe Applications for Hybrid Cloud with
IBM
and AWS
You can make that 'about 30 years ago' - time flies while we’re
having
fun.
René.
On 20 Jun 2022, at 09:51, Colin Paice<colinpai...@gmail.com> wrote:
MQ from IBM developed about 20+ years ago helped get from Batch to
real
time. You put messages to a queue, and it can run IMS transactions
(including OTMA), CICS transactions, or even batch!. You can put
on
one
member in a sysplex and get in another member.
It has single put, and also publish/subscribe capability.
Colin
--------------------------------------------------------------------
-- For IBM-MAIN subscribe / signoff / archive access instructions,
send email tolists...@listserv.ua.edu with the message: INFO
IBM-MAIN
--------------------------------------------------------------------
-- For IBM-MAIN subscribe / signoff / archive access instructions,
send email tolists...@listserv.ua.edu with the message: INFO
IBM-MAIN
---------------------------------------------------------------------
- For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO
IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send
email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
