Events in our software
More and more often we need to move data around in near real time, both internally and to external providers. In the past we used a simple long polling approach, which works fine for small amounts of data, but isn’t really for suited when you need to move thousands of events a minute. We realised we needed some kind of message broker, and we had a choice to make.
Message brokers
There are multiple services that can pass around messages like that. We immediately identified three possibilities:
RabbitMQ
Rabbit is the industry standard for message brokers where very high reliability is required and complex features are needed. It does, however, come with a lot of management overhead compared to other solutions. Since we didn’t really need any particularly advanced feature, nor we needed extreme reliability (we would need some kind of message recovery anyway), we ruled out RabbitMQ.
Microsoft Service bus
Microsoft offers a message broker by the name of service bus. It’s based on a Microsoft SQL Server instance and we already have some experience managing it. That said, our past experience with the software wasn’t particularly good. Moreover, the on-premise version is out of support and the cloud version is only available on Azure. Since most of our services today are on AWS, managing a service on a different cloud didn’t really make sense.
Redis
Redis has a couple features that can help with message brokering: pub/sub and streams. The former is a very simple to use, very basic broker. We decided we needed something more, because pub/sub doesn’t offer features like routing or the ability to read messages sent after a subscriber disconnection. Enter streams: it offers everything offered by pub/sub plus more, including the aforementioned routing and subscriber disconnection capabilities. We developed a small POC to test it out, and we ended up choosing it as our broker.
Integrating it in ISBets
When the time came to implement it in ISBets, we already had multiple uses for it. We needed it both for internal uses (sending odds changes to the frontend) and external uses (sending transactions to an external provider for bonus campaigns management). We decided to set it up as a single system that we can read internally or send to custom webhooks. When we’re moving the data internally we are reading it directly from Redis. We use consumer groups to make sure the data is replicated to each module. When sending it externally we read it through a bespoke service that identifies which third party needs which message, converts the message to the correct format and sends it to a web endpoint. What “the correct format” means depends on the third party: while we do have our own data format, some third parties require their own.
Data maintenance
Since streams don’t automatically delete their data we had to devise a system to delete data once it’s not relevant anymore. We decided for two systems: using XADD with MAXLEN and a scheduled maintenance that clears older messages with XTRIM. This lets us have a system where we keep messages sent in a set timespan and also doesn’t exceed it’s maximum size (which saves us from filling up the RAM on Redis).
Porting to iSTrust
iSTrust is a betting monitoring system aimed at governments. It used to use Service Bus to move events to generate tailor-made reports and calculate taxes. Since it needs to run on premise, we had to migrate it to Redis. We applied many of the same concepts we did for ISBets, except iSTrust doesn’t need to send data to any external service (and it likely never will), so all events are simply read internally (for example to populate reporting databases).
The long-term benefits
As previously mentioned, there are often external integrations which require us to update them in near real time about certain events. Having a webhook system backed by Redis streams has let us integrate these providers with much more ease than before. This has meant we were able to integrate a vast array of new services on our platform.