Over time, the typical business acquires more and more applications. Some will be bespoke, some off the shelf, some cloud hosted SaaS tools and some in the data centre.
Over time, business requirements arise where these need to be integrated by exchanging data between them. For instance, every time an order is placed on our eCommerce website, a customer record should be created in the CRM system, the marketing system, and the ERP system.
Next, people start to want to extract business intelligence from these systems in order conduct analysis using data from multiple transactional systems. The approach has traditionally been to “extract, transform and load” data from the line of business applications into a centralised data warehouse or data lake for the reporting and dashboards. This is essentially more data integration.
This is all standard stuff which has been keeping developers busy since the dawn of time.
There are a few long standing problems with these approaches though:
Architecturally, this all feels like a bit of a mess, with multiple point to point integrations exchanging data in large batches in an inflexible and fragile way.
In the early 2000s, people started adopting the message broker pattern, where we started putting messages onto centralised queues or topics, sometimes referred to as Message Buses, which subscribers could then listen to and process as events occured. Common technologies included Tibco, Websphere Message Broker, RabbitMQ and lots of others.
Though the messy point to point integrations described above are still very much a reality, message brokers did help to decouple systems and move away from fragile point to point integrations, particularly in complex enterprise environments.
In 2010, Apache Kafka was developed and released as an open source project by LinkedIn. Kafka gives companies similar advantages to message broker technologies in the way that it integrates systems in a decoupled and controlled manner, but has a number of benefits and key differences.
Firstly, the non functional benefits include:
Kafka is more than a just a better message broker though, and in fact refers to itself as a “distributed streaming platform” which has fundamental differences to a message broker. For instance:
So to return to the question in the title, what can Kafka do for your business?
Kafka has been deployed by over 60% of the Fortune 100 and growing, who are all securing the benefits of the above. If you would like to joint them and achieve any of the above, please get in touch for a discussion.