Sinapse with Apache Kafka: a great integration

Mases Features, News

Open source is often a source of high quality ideas and products that have numerous applications in infrastructure and services. This article describes the plugin developed to integrate Apache Kafka, a product that is part of the Apache Foundation constellation, into Sinapse.

The history and general usage of Apache Kafka platform can be read in various articles on the Web, here it is a short list: Kafka Wiki, The Log, Running Kafka At Scale.

Sinapse platform cannot miss the opportunity to interact with a product that candidates itself to be a key-product in the integration between heterogeneous systems in the near future.

This article will not speak about Apache Kafka, for this matter many articles are available on the net; what will be described will be the possibilities offered by the plugin and how they have contributed to create efficient, robust and dynamic systems.

The Sinapse Kafka plugin uses the library provided by Confluent Inc. for communication with Apache Kafka.

The library offers the ability to manage both Producer objects and Consumer objects: both of them gives the ability to manage a producer/consumer communication pattern. The data exchanged are byte streams that represent the content of messages that in many cases are JSON messages.
The Sinapse platform works by exchanging typed messages to describe the logical and functional flow.

The Sinapse plug-in for Kafka can simultaneously handle the various messages typologies: sinapse messages and JSON messages can be interpreted by the script within the plugin.

Moving to the field some practical use cases are described below.

Data processing

The Sinapse platform has numerous applications in many areas and can connect to many external systems whether they are devices, databases, filesystems, third-party products.

Inside Sinapse, Kafka can be used to integrate these systems and introduce mechanisms for analysis and processing of information that are dynamic and adapt to the needs, as well as the ability to make real-time changes and adaptations without interruption of information flows (hot-update).

The figure below shows an overview. 

Sinapse server connector

The HOSTINTERCONNECTOR article describes how the plugin creates bidirectional portals for messages between Sinapse servers instances in order to create a distributed system. The article does not describe how this is done, it is an intrinsic feature of the Sinapse platform. One of the limitations of the HOSTINTERCONNECTOR plugin approach is the non-resilience of messages (messages resilience is included only in Enterprise version) .

With the Kafka plugin the same bi-directional portal is recreated using two Kafka topics while, at the same time, Kafka provides the resilience.

The configuration is very simple: two topics are mirrored in the Sinapse Kafka Plugin. Moreover, Kafka’s resilience allows to look back at the history of events and analyze the data flow exchanged between the various servers.

The figure below shows an overview.

Other possibilities will be shown in subsequent articles.

All product names, logos, and brands are property of their respective owners.

No related posts.