blog image

Streaming Technology – Sourcing Events

In order to move towards Event Streaming and Event Based Architecture, the first thing we need to do is source data in real time and publish it onto an event stream. Even this first step can be difficult, because many applications do not “publish” events as they are entered by users. Instead, these applications interact

Read More
blog image

Free Guide – An Overview Of Streaming Data Technology

Many companies are looking to use their data more effectively in order to improve their customer experience and the efficiency of their business. Speed is an important part of this. The earlier you can respond to incoming data, the more opportunities you have to improve the customer experience. Recognising this, businesses are implementing Streaming Data

Read More
blog image

Databricks Structured Streaming Example

Spark 2 introduced the concept of structured streaming, giving users the ability to process streams of unbounded data using higher level abstractions. This is an extremely powerful capability which allows data engineers to do streaming transformations and analytics over data as it is ingested, and potentially join and integrate this with batch data at rest.

Read More
blog image

Why Real Time Analytics Is More Than Just Faster Business Intelligence

In many situations, the earlier we respond to incoming data the better.  This might be in a genuinely real time situation such as a self driving car, a trading system or a fraud check, or a more vanilla business scenario such as a product out of stock which we hope to inform our users about

Read More
blog image

Moving From Batch To Streaming Extract, Transform and Load

There are many situations in Enterprise IT where we need to move, copy or integrate data.  For example, populating a centralised data warehouse or data lake, integrating two systems such as an ecommerce and CRM system, or exchanging data between partner organisations.    Moving data in this manner is referred to as Extract, Transform and

Read More
blog image

Why Serverless Is The Future For Data & Analytics Platforms

In the bad old days, the first step when building a database, analytics or business intelligence solution would be to order or provision a number of servers for hosting.  Particularly in the data world, you would have to worry about capacity planning and over provisioning of infrastructure to account for future volumes of storage and

Read More
blog image

Recent Success With Databricks

Over the last 6 months, we’ve been working with Databricks for a client project.  For those who aren’t aware, Databricks is a SaaS/managed version of Spark, the popular open source big data processing framework. Though we were initially sceptical about Databricks and leaned more towards DIY Spark, we had a very good experience and have now moved

Read More
blog image

Materialised Views On Event Streams

Imagine we have a stream of events representing new orders: Much of the move to event driven architecture is about responding to these individual events and building analytics over them in order to improve time to insight and make the business more responsive. However, there will always be much more traditional requirements that we might

Read More
blog image

Deriving Real Time Events From Legacy Systems

As we know, a lot of business intelligence and data processing within enterprises is batch based, where periodically (e.g. daily) we exchange and process large volumes of data in batches of multiple records. Fundamentally, businesses are looking to speed this up in order to use data for operational purposes, to reduce time to insight, or

Read More
blog image

Using Real Time Data To Guide Employee Next Best Action

There has been a long held concern as to whether “Artificial Intelligence” will ultimately, replace humans en masse in the workforce. For now, this hasn’t happened, with relatively few organisations outside of Silicon Valley technology companies bringing AI & ML solutions into production, let alone to the extent they can replace large swathes of employees.

Read More