This past weekend was absolutely horrible. The building where my workplace is located were required to carry out repairs to the buildings power grid which required all electrically equipment to be turned off.
What this meant for a colleague and myself, weekend work. Unlike lots of places we don’t get paid for weekend work, there is no overtime or any other form of compensation. It’s an expectation/requirement of our jobs to be able to work on these occasions.
Knowing that this would entail the orderly power-down of our entire Data Center was a nightmare in the waiting. We arrived on premises Saturday night and over the course of a few hours worked through our power-down procedures. There were no hick-ups or surprises. To a degree we were actually stunned and quietly optimistic that for once things might finally work as they are supposed to.
I had a leisurely day at home on Sunday morning, we weren’t scheduled to start work on the power-up procedure until mid afternoon. Things were going well, small wrinkles here and there highlighting where some changes were needed in our processes. We also found some points where we could make some efficiencies and streamline aspects of the procedure.
We were nearing the end of our power-up procedures, whereby we re-instate the VDIs ( virutal desktop infrastructure ) into ready state. Our staff use thin-client desktop technology and each of these connects to a virtual computer. From the perspective of the user, it looks, feels and operates like Windows 7.
At around 6pm I started to have this sickly feeling in the pit of my gutt. This didn’t feel right, things weren’t behaving as they should. Then we hit a wall and appeared to be stuck. Roughly only a third of the desktops had started and of these we had inconsistent performance accessing them. We tried a few of our previously successful fault resolution processes and none of these helped. At this stage, Monday was going to be a bad day.
It was time to call Citrix for support. Without going into all the super boring technical aspects, citrix support were basically useless and at 1am we called it a night and went home. We had made some progress independent of their “help” which we hoped would buy us some time in the morning.
A good nights sleep and approaching the problem fresh often helps a lot. Not this time. The problem persisted and we were fast approaching a time where staff would arrive at work and not be able too, as the desktop resources were not available.
A further call was made to Citrix and another engineer to whom we had to get up to speed and what our problems were. After several more agonizing hours and no progress made, yet again, I called it a day. My colleagues stayed on. Staying on, getting progressively more tired and frustrated certainly wasn’t going to help.
I had a hunch that this problem wasn’t so easily solvable, otherwise Citrix support would have/should have been able top fix it.
Another good nights sleep and I arrived at work at 6am with a fresh perspective. From the previous day there was one lingering question we couldn’t answer. We could see the symptom of the problem and couldn’t see a common thread.
With a quiet office, nice music playing I now had the time to see what was really going on. After an hour I had found the bread crumb trail, that one important clue I needed, I hoped existed so I could find and fix this. I tried a few small scale tests and results were good. All of a sudden, it was like I could see the Matrix. Everything was clear to me. What was happening, why, all those subtle and hidden interconnections.
I worked away quietly and by the time my colleague had arrived I was in the final stages of finishing my work and was happy to answer his question of “hows the system?” with “its all fixed and back to normal”.
Finally we were done. Time to just sit and reflect. Citrix Support sadly simply isn’t up to the level they need to be to support their own product. I have no doubt it was the worst decision ever made. I have so little confidence in Citrix and their products, that if the decision were left to me I would never even mark them as a serious contender.
Time for a cup of coffee.