What do you do when your company’s main mailserver dies on you? Well most companies would probably have spare hardware to fix it. They would probably also have backups ready.
Well we had neither. The hardware failure was in the RAID-5 setup of the server. Normally you choose a RAID-5 setup so you know your data is protected. In this case, the RAID-5 setup was made out of 3 disks. Sane people would have had 5 disks for this at least. Somehow, 2 of the disks managed to die on us due to a hardware failure of some sort. It started out with the controller not being able to see one of the disks, and it ended with the controller only seeing individual disks. Good-bye array.
What happens next? Well you kind of start to panic and ask your manager if he’s made a backup (he had been busy fixing the backup for our own servers). He turned pale and looked shocked. So long backup.
Great what to do now? Well since I made a mail setup with 2 front-end mailservers to do virusscanning and spamstomping, I was able to retrieve a list of domains from the mailertable (don’t we all love sendmail?). So now we have a list a bit shy of 3000 domains, including our own domains.
The only server that had “enough power” to handle this was was our shared Plesk server. So the team decided to add the domains in Plesk and that I was the one to do this. Great(!).
That’s quite a big task, having only clicked around in plesk a little bit, I was afraid I had to add them one by one on the control panel.Then, that night, my boss calls from Russia, he was on a business trip to the people who bring Plesk to the world. He put the CEO on, who asked me my ICQ number. Ten minutes later, Plesk Support contacted me on ICQ. So a big thumbs up for the Plesk Support :)
In the end, creating the domains (2764 in total) took until 4am using the script.
The next day, all hell broke loose at the office, somehow nobody was able to get their email (what a surprise). The phones never stopped ringing for the remainder of the week.
