How Hexagonal Architecture saved my life !

Saving one year of 4 developers work in half-day thanks to Hexagonal Architecture.

The purpose of this article is to share my experience and how this event changed my mind about software development.

Context

I was working for a well-known French luxury retailer who wanted to increase its business through its e-commerce website. This website has pretty high traffic with 100,000 unique visitors per hour on special operations such as Black Friday and it represented n% of the global revenue, which was not negligible when all these retailer stores closed due to the COVID.

The website was running on a legacy stack with a monolith and we were splitting it into services such as an asset API, an offer API and so on. Therefore, there was a team working on a key API to manage the product catalog from the referential application to the merchandising indexing engine. This API has the following purposes :

• Temporarily stores the product catalog in order to send it to the indexing engine. This was a technical constraint from the fact that we had to run half of the website on the legacy monolith and the other half on the new API. So, we needed to store the data temporarily in order to trigger both new and legacy indexation at the same time.
• Triggers and monitors the indexation.
• Expose endpoints to the frontend so it can search using the indexing endpoint.

The designed architecture was as described on the following diagram:

Here the description of each phase:

1. The application mastering the catalog referential is triggered by its own cron job to read in its database the catalog data and then send them to Kafka to feed a product catalog topic.
2. The API consumes the product catalog from the topic, transforms them to its own model and then stores them in its database.
3. The API full indexation job is triggered by an event sent by the legacy monolith through Kafka to synchronously launch the legacy and new full indexation. This job reads the products and writes an in-memory zip which is then streamed to the indexing engine serveur.
4. The front-end of the e-commerce website is querying the API to get catalog pages or product pages. The API translates the front-end query to an indexing engine query and calls it to return the corresponding catalog pages or product pages.

Performance issues

After one year of project with a team of 4 developers, they deployed an implementation of this architecture and they had tremendous performance issues. At that time, they had 8 weeks to resolve these issues and finished the API to deliver it in production, because of the COVID effect on our client business which had to reduce the IT team by 50%. The delay was imposed by the incoming period where the retailer met new brands and showed them the new website to sell new products on the website and also because that was our last open date to ship in production before the commercial operations.

I asked my client to lend some help to this team and so I started an analysis of the architecture and of the performance issues in order to work with my teammate to resolve these problems. It took me one week to go through the code and run performance tests to write my analysis.

I measured the following performance issues:

• Phase 2: consuming the Kafka topic and storing in the API database took around 6 hours for 200,000 products consumed with 6 parallel API instances deployed.
• Phase 3: reading the database to write the zip and then send it to the indexer engine took around 15 hours for 200,00 products to index for an only dedicated instance of the API.
• In the legacy architecture, the equivalent of phase 2 + 3 took one hours.

Here a summary of my analysis:

• The performance issues came from the database model which was too complicated with more than 30 tables to represent products with assets, prices, stocks and so on. Spring-data-jpa was used to manage the persistence layer into the API and we all know how it can turn really bad with turnkey framework with complex needs and few skills on this framework … and this is how bad it turned for us !

• The API does not need a database as it just needs to store temporarily the data. Indeed, a master database is already existing. Moreover, the data is stored in the master database, in the Kafka topic and in the indexer engine.

At that point, the thing to really understand here, it’s that after a year of work for a team of 4 developers, we were wondering if we had to simply start over the API or if we could find a way to quickly resolve these issues and not throw away one year of work !

Solving the issues

You are reading my story for 5 to 10 minutes now and I still have not written the words “Hexagonal Architecture”. At this point of my 4 year career as a software engineer, I had already developed two API designs with Hexagonal Architecture and I understood the benefits of this loss coupled architecture, but I never experienced the use of this high modularity design.

I remember telling myself “ In which terrible situation I put myself by lending my help on this ? Removing a database to put a cache must be a tremendous refactoring and we have not time to do it”. We all know or have experienced a terrible project with high coupling between domain or business layer to the technical solutions and most of the time we run away from this kind of project.

In my head I had already chosen the storage solution: a Redis seems to be the best solution. A fast and resilient caching solution with a replicated cluster managed by GCP. Easy to deploy, quick to implement the java client and pretty cheap compared to a managed tuned Postgres database.

So, I was prepared to announce the bad news: coding the API almost from scratch and missing the due deadline …

But no! It appeared the API was following a hexagonal architecture, also called clean architecture and so in the API, we had a “database-adapter” (a maven module totally separated from the rest of the API code) and I just had to switch this adapter to a “cache-adapter” and that was it !

So, we did not need to start over and code a new API from scratch because of poor application design with a database layer strongly coupled with the rest of the API. Moreover, the business intelligence was well coded on the domain, we still keep the good and hard work of the 4 developers during one year.

Here an illustration of the hexagonal architecture with the database-adapter replacement :

It took me half a day to make a Proof Of Concept implementing a simple Redis client instead of a Postgres persistence layer and we got the following results :

• The saving time of a single product went from an average of 5 minutes to an average of 20 microseconds …
• Phase 2 took exactly the process time of phase 1 as phase 2 computation time was fastest than phase 1 !
• Phase 3 went from 15 hours to 20 minutes !

We managed to be right on time for the production with enhanced performances compared to the legacy stack.

Conclusion

So, yes Hexagonal Architecture might have saved my job during this difficult pandemic situation, because we had to be on time … But, more important, I experienced concepts I read a several times :

• Design first your domain and not your technical stack. A good architect always focuses and starts with business and finishes with technical details and implementations. The database layer was coded first and the code represents half of the API, whereas a simple in-memory store could have been done really quickly in order to deliver business value after some weeks.
• Use well known principles/designs experienced over the past decades such as Hexagonal Architecture, Design Pattern, SOLID, … They guide us and always give us tools to manage tough situations.
• Fail fast to learn fast ! Do not wait one year to deploy the first version of your application !
• Some technologies such as databases required high skills and experiences that a developer may not have. Do not hesitate to ask a DBA to optimize your DB model.
• Be careful while using frameworks. They are tools, not the essence of the software.