A mistake I made with concurrent requests

·3 min read

I was working on a feature that lets users request an advance against an available balance.

Before approving a request, the backend calculated how much was still available, checked the requested amount against that limit, and if everything looked good, saved the request and triggered a payout. The available amount was calculated by summing all the user's previous requests and ignoring the ones with a failed or rejected status. The logic looked correct, and honestly, I thought that was enough.

Then we discovered a bug I hadn't thought about. On the mobile app, there was a small UX issue: after tapping the submit button, it took a few seconds before the modal closed, and during that time the button was still enabled. Some users kept tapping because they thought nothing had happened, so the backend received multiple requests almost at the same time.

The problem was that every request executed the same logic independently. They all calculated the available amount before any of them had saved their request.

Every request saw the same balance.

Every request passed the validation.

Every request emitted a payment event.

The payment service then processed all of them, which meant some users received much more money than they were actually allowed to request.

This is basically what happened:

Request A: READ available amount → CHECK OK
Request B: READ available amount → CHECK OK
 
Request A: SAVE request → SEND payment
Request B: SAVE request → SEND payment

Both requests were correct on their own. The problem was that they were running at the same time.

The fix

After doing some research, I came across database locks. A database lock temporarily reserves a row while a transaction is running — you can think of it like reserving a table at a restaurant. Until you're done with it, nobody else can use it.

In my case, a pessimistic write lock was exactly what I needed:

await manager
  .getRepository(UserEntity)
  .createQueryBuilder('user')
  .setLock('pessimistic_write')
  .where('user.id = :id', { id: user.id })
  .getOne();

I wrapped the whole operation in a transaction and locked the user's row before doing any calculations. The flow became:

  1. Lock the user.
  2. Calculate how much has already been requested.
  3. Check if the new request is still within the limit.
  4. Save the request.
  5. Release the lock.

Now, if another request comes in at the same time, it has to wait. When it finally gets the lock, it recalculates the amount using the latest data in the database, including the request that was just saved. That means the second request sees the updated balance and gets rejected if the limit has already been reached.

The same situation now looks like this:

Request A: LOCK → READ → CHECK OK → SAVE → UNLOCK
Request B: WAIT → LOCK → READ UPDATED DATA → CHECK FAIL

Besides the database lock, we also made other improvements. The submit button is now disabled while the request is being processed, and we added more safeguards to make this kind of issue much harder to happen again.

What I learned

The main lesson for me was simple: if a write depends on data you just read, you need to think about what happens when two requests read that data at the same time.

· · ·