In order to move towards Event Streaming and Event Based Architecture, the first thing we need to do is source data in real time and publish it onto an event stream. Even this first step can be difficult, because many applications do not “publish” events as they are entered by users. Instead, these applications interact
Read More
Many companies are looking to use their data more effectively in order to improve their customer experience and the efficiency of their business. Speed is an important part of this. The earlier you can respond to incoming data, the more opportunities you have to improve the customer experience. Recognising this, businesses are implementing Streaming Data
Read More
Spark 2 introduced the concept of structured streaming, giving users the ability to process streams of unbounded data using higher level abstractions. This is an extremely powerful capability which allows data engineers to do streaming transformations and analytics over data as it is ingested, and potentially join and integrate this with batch data at rest.
Read More
In many situations, the earlier we respond to incoming data the better. This might be in a genuinely real time situation such as a self driving car, a trading system or a fraud check, or a more vanilla business scenario such as a product out of stock which we hope to inform our users about
Read More
There are many situations in Enterprise IT where we need to move, copy or integrate data. For example, populating a centralised data warehouse or data lake, integrating two systems such as an ecommerce and CRM system, or exchanging data between partner organisations. Moving data in this manner is referred to as Extract, Transform and
Read More
In the bad old days, the first step when building a database, analytics or business intelligence solution would be to order or provision a number of servers for hosting. Particularly in the data world, you would have to worry about capacity planning and over provisioning of infrastructure to account for future volumes of storage and
Read More
Over the last 6 months, we’ve been working with Databricks for a client project. For those who aren’t aware, Databricks is a SaaS/managed version of Spark, the popular open source big data processing framework. Though we were initially sceptical about Databricks and leaned more towards DIY Spark, we had a very good experience and have now moved
Read More
Imagine we have a stream of events representing new orders: Much of the move to event driven architecture is about responding to these individual events and building analytics over them in order to improve time to insight and make the business more responsive. However, there will always be much more traditional requirements that we might
Read More
As we know, a lot of business intelligence and data processing within enterprises is batch based, where periodically (e.g. daily) we exchange and process large volumes of data in batches of multiple records. Fundamentally, businesses are looking to speed this up in order to use data for operational purposes, to reduce time to insight, or
Read More
There has been a long held concern as to whether “Artificial Intelligence” will ultimately, replace humans en masse in the workforce. For now, this hasn’t happened, with relatively few organisations outside of Silicon Valley technology companies bringing AI & ML solutions into production, let alone to the extent they can replace large swathes of employees.
Read More