Open source is often a source of high quality ideas and products that have numerous applications in infrastructure and services. This article describes the plugin developed to integrate Apache Kafka, a product that is part of the Apache Foundation constellation, into Sinapse.
This article will not speak about Apache Kafka, for this matter many articles are available on the net; what will be described will be the possibilities offered by the plugin and how they have contributed to create efficient, robust and dynamic systems.
The Sinapse Kafka plugin uses the library provided by Confluent Inc. for communication with Apache Kafka.
The Sinapse plug-in for Kafka can simultaneously handle the various messages typologies: sinapse messages and JSON messages can be interpreted by the script within the plugin.
The Sinapse platform has numerous applications in many areas and can connect to many external systems whether they are devices, databases, filesystems, third-party products.
Inside Sinapse, Kafka can be used to integrate these systems and introduce mechanisms for analysis and processing of information that are dynamic and adapt to the needs, as well as the ability to make real-time changes and adaptations without interruption of information flows (hot-update).
Sinapse server connector
The HOSTINTERCONNECTOR article describes how the plugin creates bidirectional portals for messages between Sinapse servers instances in order to create a distributed system. The article does not describe how this is done, it is an intrinsic feature of the Sinapse platform. One of the limitations of the HOSTINTERCONNECTOR plugin approach is the non-resilience of messages (messages resilience is included only in Enterprise version) .
The configuration is very simple: two topics are mirrored in the Sinapse Kafka Plugin. Moreover, Kafka’s resilience allows to look back at the history of events and analyze the data flow exchanged between the various servers.
The figure below shows an overview.
Other possibilities will be shown in subsequent articles.
All product names, logos, and brands are property of their respective owners.