Joshua Paling

Breaking Prod

When I’ve broken production apps, stakeholders have often asked: “What can we do to ensure this doesn’t happen again?”

Bugs vs speed

The relationship between development speed and bugs looks something like this:

As you slow development speed, the impact of bugs does not decrease linearly

Rushed pace & poor testing means many bugs. Slow down, and at first you’ll drastically reduce bugs. But slow down more, and you begin to bottom out. You can never reach zero.

Breaking prod less means using time, money, and bureaucracy to slide right on an axis of diminishing returns.

Finding the sweet spot requires ongoing collaboration between business and tech. Since sliding RIGHT is expensive, you want to be as far LEFT as your appetite for risk allows.

Writing pacemaker firmware? Slide very far right to ensure no one dies.

Selling shoes online? You have a higher tolerance for risk. Slide left and take advantage of it.

Coming back to the original question, the answer is “we can slide as far right as you want, but it’s not free… how much do you want to pay for it?”