Event Driven Architecture - the story of how we build Microservices

Nowadays Microservices are probably one of the most commonly used buzzwords in the whole tech world. But what does it actually mean? How does it help us to continue our mission of improving financial life? What is Event Driven Architecture and how does it help us to scale a Microservice infrastructure? In this deep dive we will try to give you the answers!

What are Microservices?

There are a lot of definitions, but the easiest way to understand what Microservices stands for is probably by comparing it with a traditional monolithic architecture.

On one hand, a traditional system designed as a monolith. It contains all modules needed to create an application. Therefore, this is a single unit that is deployable at once. It’s easier to build, maintain, debug and test, but it also has some limitations.

On the other side, we have a distributed system with a bunch of smaller Microservices. Each of them performs a part of a domain logic in order to create a system. It looks much more complicated - and to be frank, it is. On the other hand, it gives us new and great possibilities, provided that it’s implemented well. The goal is to keep these Microservices loosely coupled and as independent as possible. This will enable:

Better scaling - the load across the parts of the system might be various. Since each of them is a separated unit, we can quickly scale it based on the current traffic. It also makes our architecture more cost effective.

Increased reliability - the potential outage on the part of a system should not affect the other components. With Microservices, the risk of a total failure is lower because they work as separated units with their own error handling mechanisms. Thanks to that, only some functionalities are temporarily unavailable rather than the whole system.

Technology independence - we are no longer forced to choose a single technology for our system. Each microservice can be created with the best suited programming language, framework and storage that matches our expectations in a particular case.

Structured teams - each team is able to focus on a certain part of a domain and its corresponding Microservices. It is also easier to be onboarded into the company because you don’t need to learn the whole system at once. Not to be underestimated in a growth company.

Replaceable components - each internal change in any service does not affect a whole system, as long as the external contract is fulfilled. Thanks to this, we can easily upgrade, refactor and change service implementations.

As you can see, these benefits enable us to build better, more efficient and highly available products for our customers. It’s also worth mentioning some potential disadvantages and challenges that may occur in a distributed system and how to cope with them:

More difficult testing and debugging - verifying whole functionalities in distributed systems requires additional competencies and techniques, especially if it’s written in an asynchronous way. When some functionality fails it’s harder to trace and find the root cause. Such techniques, like correlationId, need to be implemented in order to keep Microservices traceable.

Network instability - Microservices communicate through various network protocols. Unfortunately, it’s not stable like inside-process communication in monolith applications. It means we should always assume that something might fail. Some resilience patterns, such as retries or fallbacks, should be implemented.

New competencies needed - working in distributed systems requires new skills and tools. DevOps mindset is needed across the teams in order to create highly automated environments with proper logging, monitoring and CI / CD process.

Eventual consistency - using techniques like sharding, clustering, read models etc. cause data inconsistency for some period of time. It results from Cap Theorem. This is something that must be discovered and managed individually depending on the specific business case.
The decision to start building applications in a Microservices architecture is crucial. The advantages show that sometimes this is the only way to go if you want to be competitive in the market. On the other hand, there are additional costs and efforts needed. That makes the decision worth thinking about and evaluating properly. Below is a list of things that if you answer “yes” is a strong indicator that you should move to Microservices:

Your code base / team becomes too big for just one solution

You want to scale your organization

You want to be more agile

You want to introduce fully autonomous teams

Your market is changing rapidly

You are capable to make architecture investment
It’s worth keeping in mind that you probably shouldn’t start a new project in Microservices Architecture. It’s hard to divide a domain properly without the domain's discovery. Using techniques like Event Storming, combined with Domain Driven Design, is very helpful. At this stage it’s nice to start with a modular monolith and be ready to quickly extract a module into Microservice when needed.

Our case study

As you may have heard, at Northmill developers work closely with product managers in order to bring the best products for our customers. Before we start implementing new functionalities, we discuss them during several refinement sessions. It doesn't mean that we can just get business requirements and code them as described. Our responsibility is to design it based on the specification, but also improve the overall quality of the system. We can achieve that by choosing the right patterns and architecture depending on the individual case.

Typically, business logic requires performing several operations. Sometimes it needs to be sequential, but other times it is possible to run it asynchronously.

Not long ago, we were challenged to implement reporting of the loan status changes to our partner. We checked the code and noticed that similar functionality had been done before.

It sounds pretty straightforward because it’s fully synchronous. It means operations are executed one by one. In this particular business case, we don’t need to wait for all responses and block the user interface. But it also has its disadvantages:

Payout service is blocked by other dependencies that might be done in the background

Payout service is not extensible. We can imagine that there will be a lot of new actions to execute after payout and the total operation would become enormous

Microservices are not independent. Single failure in this chain disrupts the whole process

Microservices are not decoupled. The payout service needs to have knowledge about other domains like Emails, PDF, Partner integration.

Probably you have already noticed that in this example we’ve lost the biggest advantages of Microservices architecture, along with the exact purpose and sense of its use. This solution continuously becomes less reliable and fault tolerant. High levels of mutual dependencies would escalate quickly. Based on this analysis, we’ve decided to make some investment and design this process in an asynchronous way.

Improved version

The decision was clear - it was a perfect business case to try Event Driven Architecture. The idea of this approach is to set up communication between the parts of the system, through events in an asynchronous manner.

We can distinguish two main roles - a producer who sends an event to the event bus and a consumer (one or many) who asynchronously processes the message. Event informs about a change of state that happened in service. It contains some data that might be used by other services in order to perform their part of domain logic.

The difference is that the Payout microservice doesn’t invoke other Microservices directly. It publishes an event called ‘LoanPaidOutEvent’ to the event bus. Then this event is received and processed by subscribed Microservices.

This way, we have solved the main problems of the synchronous solution described earlier:

Microservices are decoupled. Payout microservice doesn’t have knowledge about other microservices. It just publishes events to the event bus and doesn’t wait for any response.

Microservices are independent. Payout microservice is less sensitive to any consumer’s failure because events are processed asynchronously. This way errors can be handled individually by each microservice, minimizing impact on the other parts of the system.

The system is more extensible. We can quickly add other consumers without touching the producer's code.

The solution becomes more responsive since we don’t block the main thread and perform some operations in parallel.

Event buses overview

This infrastructure component is an intermediate layer between Publishers and Subscribers in the Event Driven Designed system. Nowadays, there are many of them available offering various features. The most commonly used are RabbitMQ, Kafka, AWS SNS, AWS EventBridge and Azure Service Bus. In the following articles, we’ll describe how we use some of them.

It’s crucial to choose the most suitable one for your system. These factors might help you make the right decision:

Messages ordering - imagine two events - LoanCreated and LoanCanceled. It’s unacceptable to process these messages in the wrong order since it may possibly corrupt your data.

Delivery guarantee - ask yourself what happens if a message never gets delivered, or is delivered multiple times? Offering “at least once delivery” is a standard. Combined with idempotent handlers might be enough. Some providers also guarantee “exactly once delivery”. If you don’t want to rely on a provider you should use patterns like Inbox and Outbox, in order to gain “exactly once processing”.

SDK programming languages - make sure that SDK for your programming language is available and offers all the features you need. Some of them might be limited or have expired support.

Scalability - you will probably require high throughput and ability to scale. Providers bring various mechanisms like partitioning, replicas. Serverless solutions are also worth attention.

Reliability - verify that messages can survive crashes. Usually, they are securely persisted in storage. It’s also nice to have some retry mechanisms and other fallbacks in case subscribers are temporarily unavailable.

Management method - you can choose self hosted event brokers that you need to set up and maintain on your own servers. On the other hand, you can go with SaaS solutions that are managed by a provider, but are also more expensive.

Summary

There is no single recipe to build Microservices correctly. I hope that I’ve convinced you to treat it as a powerful but challenging way to build modern IT systems. It requires a change of habits and use of different techniques. Event Driven Architecture helped us to improve Microservices in the particular business case, but it doesn’t mean it's useful everywhere. Every architectural decision has some consequences. Let’s choose the architecture for the problem, not the problem for the architecture.

Paweł Kiełkowski
Backend Engineer