On 7/6/2023 4:00 am, Frank Swarbrick wrote:
Does anyone publish messages to Kafka from z/OS?
Yes https://www.ibm.com/docs/en/om-im/5.6.0?topic=introduction-architecture
If so, what technology do you use?
We use a combination of technologies in our core product. We utilize the Java Producer API and have implemented the librdkafka library, along with a CELQPIPI bridge for calling it from Metal/C (HLASM). Additionally, we employ Python for developing test harnesses. While Kafka can run on z/OS, we deploy it on a k8s cluster running on Linux VMs for better performance. Linux offers advantages such as faster execution due to zero-copy optimization, which is not available in USS.
Kafka is a remarkable technology widely adopted by mainframe sites. It empowers the creation of an event-driven architecture, leveraging Apache Kafka as the central nervous system. This architecture enables applications to communicate through events, which represent significant occurrences or updates in the system.
Within this architecture, Kafka serves as a scalable and fault-tolerant event streaming platform. Producers generate events and publish them to Kafka topics, while consumers subscribe to topics and asynchronously consume the events. This decoupled communication allows for the independent evolution of components and real-time processing of events.
The benefits of Kafka event-driven architecture include loose coupling between components, real-time processing and analysis capabilities, and fault tolerance through durable event storage. It finds diverse applications in microservices, data integration, real-time analytics, and event-driven systems. Overall, it provides a scalable and flexible foundation for building responsive and high-volume systems.
Kafka is a batch killer! Down here in Australia the government created a new payments platform using the concept of a payID (email address, phone number etc) to processe payments and transfers in real time. All the banks used Kafka as the core technology along with CDC solutions running on z/OS. Real-time settlements are becoming a must for the banks. Nobody wants to wait for overnight batch to process settlements.
Kafka offers more than just being a message broker like MQ. It includes powerful features like the Streams API and ksqlDB.
Kafka Streams is a client library in Apache Kafka for building real-time stream processing applications. It allows developers to process and transform data streams directly within the Kafka ecosystem, without external dependencies.
Key features of Kafka Streams include stream processing with operations like filtering, mapping, aggregating, and joining; support for stateful processing and exactly-once processing semantics; interactive queries to access state stores; seamless integration with the Kafka ecosystem; and scalability and fault tolerance through distributed coordination. One popular use case for Kafka Streams is anomaly detection
ksqlDB is an open-source event streaming database that integrates with Apache Kafka, providing a SQL-like query language for real-time stream processing. It supports time windows for partitioning and analyzing streaming data based on specific intervals. With sliding, tumbling, and hopping windows, you can capture events within these intervals.
By using time windows in ksqlDB, you can perform calculations, aggregations, filters, and joins on data within defined time intervals. This enables real-time trend monitoring, analysis of data in discrete segments, and capturing events that span multiple windows.
ksqlDB also offers materialized views, which are precomputed and continuously updated views derived from one or more streams or tables. Materialized views improve query performance by avoiding redundant calculations and joins, particularly useful for frequently accessed data.
Leveraging time windows and materialized views in ksqlDB empowers real-time analytics, pattern identification, and data-driven decision-making based on time-related criteria within your streaming data. It serves as a powerful SQL engine built on top of the Kafka Streams API.
One compelling use case to explore is the modernization of a performance analyzer product. The current solution relies on batch processing, where customers schedule jobs to read SMF records, perform aggregations using time windows, and generate reports with metrics like MIN, MAX, AVG, STDDEV, and more. To investigate outliers, customers can run list reports to examine individual transactions. It may take several hours before problems are detected.
By leveraging the capabilities of ksqlDB and a real-time SMF stream, we can greatly simplify this process. With a SQL query utilizing time windows and aggregate functions, we can conduct real-time analysis on the streaming data, promptly detect outliers, and emit anomalies to a dedicated topic. These anomalies can then be processed by a service responsible for sending alerts or taking additional actions.
Furthermore, the aggregated events can be directed to another topic, which can be consumed by popular analytics platforms like Elastic, Splunk, or other consumers such as Prometheus exporters. This allows for the ingestion of summarized data into an analytics ecosystem, facilitating further analysis, visualization, and integration with metrics from distributed systems.
The significant advantage of this approach is the offloading of computationally intensive tasks from traditional batch processing, resulting in optimized performance and resource utilization. This real-time processing capability, combined with the potential integration of AI/ML capabilities, signifies the next generation of advanced performance analyzers.
https://docs.ksqldb.io/en/latest/reference/
---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email tolists...@listserv.ua.edu with the message: INFO IBM-MAIN
---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
