We just had something happen that caught us by surprise. We were log shipping to a remote warm standby server. It was all running smoothly and we were ready for anything. Then the remote server went down – right down. Looks like we might have to rebuild it. Meanwhile, I’ve got transaction log backups piling up and nowhere to ship them to. Without a standby server that’s on the same backup cycle as the live server we just don’t have a DR plan that applies. Now I have to tell people I can’t do those urgent things I promised to deliver because I have to sort this out. And I thought it was hard enough getting buy-in for setting this up in the first place!
When we drew up our plan we did briefly consider what would happen in this scenario. We were far too casual. We said to ourselves, “Well, we didn’t have this capability at all until now and we’ve never had a disaster so it won’t be such a problem to be without it again while we fix it. After all, you can only plan for so much and after that you just have to wing it.” Wrong answer! The last thing you want to be doing when things are all screwed up is to start improvising. A standby server is just as likely to fail as a live server, maybe even more likely. After all, you keep a keen pro-active eye on your live servers – you are doing that, right? – but it’s hard to justify spending too much time monitoring a standby server. As long as the replication/mirroring/log-shipping works, that’s fine. Wrong! A standby server is just as important and warrants just as much care and feeding.
Don’t let this happen to you. What does your plan say about the standby going down? How does it affect your routine DR jobs etc. How will you get a standby server back again? Do you need to alert the business that you may need some budget? How will you re-synch it with your live server so you can resume normal operations again? Knowing in principle is not enough. Write it down. Test it. Know it will work.