On critical bugs and infrequent major updates

A critical bug – solved

We’ve been getting occasional reports from Poker Copilot 5 users that the HUD is not updating for long periods. Until now we’ve been unable to reproduce it.

Then two days ago, a set of coincidences let me experience the bug. The HUD would show after one hand but then it wouldn’t update. The hardest part of fixing bugs is reproducing them. Once you’ve reproduced a bug, fixing it is usually straight forward. Time consuming possibly, but straight forward.

Here’s the bug: if you have a recently created or modified file in one of your hand history folders that ISN’T recognized by Poker Copilot as a valid hand history file, then any live hand history file subsequently detected by Poker Copilot only gets handled once. Future updates are ignored until you restart Poker Copilot, or one hour has passed since the file causing the problem was last changed.

So yesterday I fixed the bug. There will be a Poker Copilot release later today that includes the fix.

I wanted to know how this critical bug got into Poker Copilot. It turns out that I added it in May, 2014, in the early days of creating Poker Copilot 5. It was a small optimisation needed for our work in making Poker Copilot capable of running on Windows.

May, 2014…and I’m writing in March, 2015. In that period I’ve had a dedicated part-time QA assistant working for months testing Poker Copilot thoroughly, many other bugs have been discovered and fixed,  we’ve gone through a long beta period, we released Poker Copilot 5 two months ago, and this critical bug managed to keep hiding.

This makes me ask myself: what could have prevented this?

•  Unit testing? I don’t think so. Poker Copilot already has hundreds of unit tests. The component where this bug occurred is not a natural candidate for unit testing. It is not good to make unit tests complicated and slow. When unit tests run slowly it becomes tempting not to use them frequently. I think that adding a unit test that would pick up this bug would be too convoluted. Unit tests, in my opinion, are best used to test black boxes, and this critical bug was in code that doesn’t easily lend itself to be a black box.
• Code reviews? In a code review session, another developer would have looked at my change back in May, 2014, and would have quickly spotted the bug. It was an egregious bug. But as the sole developer in my company, I don’t have the luxury of code reviews. I wish I did.
• Shorter release cycles? Many months passed between me creating the bug and fully releasing the code. I believe that if I had released a Poker Copilot a week or two after adding the bug, I would have been able to connect the “my poker copilot is broken!” feedback to guilty code changes much quicker. Customers would soon report the problem started with the latest minor update, I’d look at what changed in the latest minor update, and would have been able to narrow down the problem with too much trouble.

This bug isn’t alone. A month ago I discovered another long-hidden performance problem. Despite accurate reports from customers, I only managed to reproduce it by getting a helpful customer to supply both his Poker Copilot database and his Poker Copilot preferences file. That helped me spot the problem – again, the offending code had been part of Poker Copilot 5 for many months, making it through near daily testing for months, through the beta period, and into our actual release of Poker Copilot 5.

Software with infrequent major updates

The more I experience problems with major updates, the more I believe it is time to end for good the 12- to 24-month major upgrade cycle that desktop software traditionally uses. Releasing many small upgrades frequently is much safer than releasing a huge upgrade seldomly.

This is is no remarkable observation; since 2010, Google Chrome has six-week release cycles. Mozilla Firefox switched to six-week release cycles in 2011. Only three years ago, Facebook announced a four-to-eight week release cycle for its mobile apps. Now it seems to be every two weeks. From their iOS app description: “To make our app better for you, we bring updates to the App Store every 2 weeks.”

On the other hand, Apple releases OS X yearly. As a developer I’m always amongst the first to update to the new release, and it is always initially a painful process.  Things that used to work stop working. (For some months OS X Yosemite turned my lovely Sennheiser wireless headphones into expensive dust collectors; and my Apple TV became more trouble than it was worth to use from a Mac with Yosemite. Apple rectified these issues over some months.) Parallels Desktop, software I rely on for running virtual machines – an important part of modern cross-platform, cross-version software development and testing – released an update some months ago that worked worse than the previous version – but required me to pay an upgrade fee. I had to pay money to have what seemed an inferior product. Parallels Desktop 10 did have some helpful new features, but all I could see at first was that the new update was worse for me than the old one. Eventually they released a couple of minor updates and the problems were solved.

The difference between Chrome, Firefox, and Facebook on one hand, and Poker Copilot on the other, is, well, a lot. Two major ones for the sake of this article: they have many more resources; and their products are free. So their release cycle doesn’t have to take into account how frequent free releases rather than an annual or twice-yearly paid update affect sales and profitability.

Poker Copilot is a paid product. After a few years on the market, upgrade fees become a significant source of revenue for desktop apps. This is why having a occasional major upgrade of Poker Copilot is important for my business.

Yet customers expect significant extra value for the hassle and expense of major upgrades. So it is tempting to develop for many months, and then release a major upgrade. At release time, we can say, “Look at these new features if you pay to upgrade!” But, as my experience has shown, this leads to software that has more defects than is the case with many minor updates. As I noted with Parallel Desktops, some Poker Copilot 5 customers could say at first: I had to pay money to have what seemed an inferior product. I believe Poker Copilot 5 to be a big step forward, but for customers who were affected by initial bugs, and who didn’t need or care for the new changes, all they could see is that it brought them problems. This is a stressful situation for me; it isn’t pleasant to be trying to deal with this. The bugs causing customers grief are hard to find. While finding them, I’m not able to work on new features nor on other products. Nor rest on a sunny beach, which is always a pleasant alternative.

One way to solve this is to create subscription-based services, either for desktop software, or for online web applications. People pay a monthly- or annual- fee. While they keep paying, they get to keep using the software. When they stop paying, they can no longer use the software. Instead of major updates to get customers to pay upgrade fees, the company releases frequent small updates, to keep the customers subscribing. That is not a path I intend to follow at this stage, although it certainly is an option worth investigating.

Another way is to state clearly that a purchase of desktop software entitles the customer to the current version and all upgrades for a time period, say one year. After that time, the customer can still use the version they have, but they can no longer update to new versions without buying an upgrade license. I believe this to be more customer-friendly; the customer doesn’t feel they are forced to buy upgrades. They only do so if they believe there is value for them in the new major upgrade.

A friend of mine reckons that it is inevitable that one day Poker Copilot – and indeed most desktop software – will become an “in-the-cloud” pay-by-month service. If that is the case, I think we are still a long way from that day. Due to the massive databases that some Poker Copilot customers have, and the processor demands a database-heavy product places on a computer, I’d like to avoid that change for as long as possible.

For now, after the headaches in the last 2 months of fixing the critical problems introduced in Poker Copilot 5, all I can think is: No more major Poker Copilot updates ever; frequent minor updates from here on. The end product is the same; the path to get there is more pleasant for me and for my customers.