I once worked with a team that was building a video streaming app for users. The app followed a freemium business model where some video content was available to the users at no charge while other premium content was available to users after purchasing a subscription.

One evening, the customer service team started receiving multiple calls from users with a premium subscription complaining that they couldn't access the premium content. This was a very big problem. The customer service team escalated this issue to our software engineering team.

One person on our team checked the recent changes, another looked at logs carefully, and another tried to reproduce the issue. We could reproduce the issue! We identified a backend change that was most likely the cause for this and reverted it. After the revert, we couldn't reproduce the issue again and it seemed to be fixed.
We also asked the customer support team if the complaints had stopped and they confirmed. "Great, we fixed the problem for now", we thought.

Like many other engineering teams, we scheduled a post-mortem discussion to analyse the root cause of the issue. In this discussion, we highlighted that about 300 users were affected by it for about 30 minutes. We then came up with different ideas and actions that we needed to take to prevent such issues from happening again in the future.

Most of our action items were around code and process changes while building, testing, deploying and monitoring the release of our products. We were happy with the outcome of the meeting and felt confident about our plan.

Since this incident affected users in one of the top tiers, some of the business stakeholders were interested in understanding the outcome of our post-mortem discussion. We informed them about the actions we were going to take to avoid such issues in the future.

One of the business stakeholders, said, "You know best about software development and I'm sure you're taking good measures to avoid this in the future". He took a pause, looked around at the table and continued, "But what are you going to do right now to delight the customers that were affected by this"?

Everyone was silent. We glanced at each other and none of us had an answer.
No one had thought about what we as a software engineering team could do to delight the affected customers. We had fixed the problem the customers had experienced by reverting the code, and we were mainly focused on coming up with a plan for not having such issues again in the future. However, we had not thought about what we could do for the customers that were affected by this issue.

"Based on our budget, we could waive the user's subscription charges next month", said someone. Another said, "Instead of waiting a full month to give users something, we could credit the amount into their account today. It will signify prompt action from our end". Another said, "Instead of cash or credits, we could enable the top-tier download feature (which their tier currently did not have access to) for a month. They get a great feature for a month and maybe they will like it enough to pay for the next tier from the following months".

Many people shared ideas. Some of the ideas needed some additional development effort and some others didn't. The team, along with business folks, decided on the best path forward and planned the next steps for it.

This was a great lesson for all of us on the software engineering team that day. It's not enough to just fix issues and find ways to avoid them in the future. It is also important to think about how we can apologise to the affected customers and how to delight them.

Note: This is a fictional story inspired by some real-life events.