A time bomb reportedly left over from the beta of VMware’s two-week-old update to ESX 3.5 and ESXi 3.5 (Update 2) caused product licenses to expire yesterday.

It also happened with a patch to ESX 3.5 or ESXi 3.5 Update 2.

The experience affected a whole lot of people, apparently enough to crash VMware’s knowledge base.

VMware put out a freebie ESXi hypervisor on July 28 as a way of coping with Microsoft and its free Windows Server 2008-embedded Hyper-V.

As a result of the oversight, VMware virtual machines that were turned off couldn’t be turned back on; virtual machines that were suspended couldn’t get out of suspend mode; and virtual machines couldn’t be migrated with VMotion.

All in all it wasn’t a good day for VMware – whose new CEO Paul Maritz found himself posting an open letter of apology to the VMware web site shortly before midnight New York time last night.

It also wasn’t good for the company’s QA people who left the cursed code in the releases. They are now threatened with a long session in the virtual stocks.

“We failed in two areas,” Maritz’s letter says, “not disabling the code in the final release of Update 2 and not catching it in our quality assurance process.”

“This incident.” he said, “has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.”

VMware had to deny that the issue had anything to do with a security problem.

Maritz posted his letter after the company issued a temporary so-called “express patch” – two of them actually, one for ESX 3.5, one for ESXi Server 3.5 – and he promised “within the next 24 hours, we also expect to issue a full replacement for Update 2.”

VMware said simply resetting the host time to a date prior to August 12, 2008 had “a number of very serious side effects that could impact product environments”; anyway that patchwork workaround wouldn’t meet governance requirements.

The bug cost users at least 20 hours of downtime, not to mention the dye to cover their newly gray hairs and the effort to put everything back together again.

It also happened just as a lot of servers were being rebooted because it was Microsoft’s Patch Tuesday.

The incident also made them think about deploying multiple hypervisors, not something VMware wants to hear.