It seems we are quite sporadic bloggers. Plenty has been going on; we just haven’t taken the time to upload pictures or to write about it. Perhaps soon. Until then, Merry Christmas!!
More Gob and Lucy November 23, 2008
These pictures are a little old, but I was looking through them and realizing just how little they used to be. We’ll post recent pictures soon!
PNCB October 4, 2008
Since Brian posted about his non-trip (hopefully trip to be) I thought I would share about the trip I took in August. Shortly after I graduated I began volunteering for the pediatric nursing certification board as an external item writer. I was thrilled to learn that I was invited to attend a workshop on writing test questions – and I was even more thrilled to learn that it would be paid for! I am definitely a big nerd in that I really like to study, especially when it is on my own terms and timeline, so writing test questions in my spare time is actually a lot of fun. It also keeps me on my toes on topics in primary care.
I headed off for Reston, VA for the conference with nervous excitement about traveling alone. I was to navigate 3 airports going to and from the conference. The first part of the trip was a breeze. I had a non-stop flight to Washington D.C. then caught a shuttle to the hotel we stayed in (very nice btw). The hotel was in the middle of a large shopping area in a nice town about 10 minutes away from the Dulles airport. I had brought several books and was really excited about watching cable but was very disappointed to find that apparently the hotel signed up for all-sports tv – no food network, tlc, MTV, vH1, bravo – nada. Nonetheless I had plenty of reading material, so after grabbing some dinner I sat down to enjoy a good book. Then the power went out.
Not just a little power outage, but a full city block. I went out in the hallway to see if anyone else knew what was going on and heard the maids yelling in another language, then the fire alarm started sounding and an intercom said to exit the building, so I took that as my cue to head downstairs. On the way down I befriended another visitor of the hotel who happened to be in the navy. We went outside into the plaza outside the hotel and waited for about 2 hours with other people in the area for the lights to come back on. Several rumors swirled about the cause of the outage, but the true reason remains a mystery. All I know is that it’s hard not to be a little freaked out when the power goes out in the whole city you’re in all alone that happens to be just a few miles from the nation’s capitol. Thank goodness my freak-out threshold has been raised over time or I would have been a mess. I actually had a decent time talking with the guy I met and some of the other people waiting outside. And it turned out that the guy I met was a Christian which was cool.
The workshop was really great. I had another small freak out moment when one of my flights for home was severely delayed and I thought I’d be spending the night in an airport in New Jersey, but once again God totally helped me out and I got a direct flight to Nashville, and got home 4 hours earlier than I thought I would. Overall a great experience. An added bonus, I got to fly in a tiny propeller plane!
Production Outage October 1, 2008
Last night I took down about 6 servers for work. Important ones, too. It was supposed to be just a small fix to the zoning on our main SAN switches for something unusual that happened yesterday during the day (human error caused by a new team member who I also didn’t stop at the time). The original problem wasn’t a huge deal, but it created a small mess than needed to be cleaned up.
Apparently, I committed the fix wrong while cleaning it up later that night. It’s really easy to do. I’m kind of surprised it doesn’t happen more often. There’s an A-side switch and a B-side switch (for redundancy), and if you accidentally activate changes on the B-side with the A-side selected, bad things happen. And they did. Every server lost connection with the B-side of our storage network. Fortunately, I only made the mistake on one side, which means everything stayed connected on the A-side, so most servers could still see their storage but just had fewer “paths” to get to it. Unfortunately, not all servers are configured correctly to handle losing paths (or worse, still aren’t even connected to both sides). Those were the ones that went down. Thank God it was only one side. Accidentally bringing down all 200 physical servers and another 400 virtual servers would have been a total disaster, even after business hours. I try not to think about it, but the fact that it’s that easy for me to do something like this is scary to say the least.
There’s a root cause meeting going on now. No doubt the production control police will be grilling my team lead as to why there was no change ticket for this ‘change’ when it really wasn’t even supposed to be a change at all. It hasn’t been a very good couple of months for infrastructure. We’ve had a lot of stupid outages lately. This does not help my team out at all. The demand for only touching production devices on Saturdays will definitely rise. Those meetings can be extremely annoying, too. When something like this happens, you get to try and explain what happened to the production control team, people who generally have no clue what you’re talking about and seem to think that “if only you had created a ticket” or “if only this had been done on a Saturday” nothing would have happened. The truth is that outages are going to happen eventually. People make mistakes, and hardware fails. Saying “we must have 24/7 availability” does not instantly make it so, especially when you don’t wish to pay for that kind of environment and also expect your employees to work terrible hours and pretend that it’s “just part of their job.” It may be part of the job, but the way infrastructure employees work around here goes far beyond what I consider their core job, especially when compared to the rest of IT. I have managed to dodge most of the after-hours work on this team due to most of my contributions being scripting; most others have not been so lucky.
Things like this make me question a future in the IT department, at least on a lower-level infrastructure team. I’m not a nights and weekends kinda guy when it comes to my job, especially if I’m not getting paid overtime and/or a shift differential of some sort. Back in the day, IT apparently used to get overtime here as well as reimbursement for being forced to travel between two data centers, but those days are long gone for most all companies now because they became “too expensive.” Instead, we’re supposed to take “comp time” where if you work a Saturday, you take off the next Friday or something. The main problem is that pretty much no one ever takes their full share of comp time, usually because they have too much crap going on during the day to just take off. Another problem is that having some random day off during the week does not make up for losing a Saturday. When I have Saturday off, I can hang out with my wife and other friends/family. When I have Friday off, I sit at home by myself doing nothing while everyone else is at work.
It isn’t that I’m not paid well, it’s that I’m paid the same as other people on application teams who almost never have to wait until after-hours to actually do their work. Those teams also deal with a much lower degree of personal risk. If they make a mistake, at worst they probably just screw up their one application. If I make a mistake, I can easily bring down the entire IT infrastructure of my business unit. Managing millions of dollars of SAN equipment has been very satisfying. It really is cool stuff. But it touches almost every single server (hundreds and hundreds) in our environment. Centralized storage makes things much more efficient, flexible, and generally more awesome. It also means any outages to your SAN are absolutely devastating. When servers lose their storage, they basically fall over and enter a coma which can only be remedied by the server teams very tediously logging into every single one and rebooting. That’s assuming no data corruption, which takes a lot more work to fix. It’s also assuming people remember how to bring up a server+application which might not have been rebooted in over a year. Oh, and if an outage does happen during the day and some of the really important stuff goes down, God save you from the wrath of everyone else in IT because you just negatively impacted a very important part of their quarterly bonus. People get angry when you’re the reason they get paid less.
I’m sure the idea is that everyone in IT is equal and no one should be treated differently. It’s just not true. Some jobs are harder or more important than others. Having worked on one of if not the most critical team, I can say that it’s just not worth it if there isn’t some other incentive. Why would I want to work on a team that has to do most everything after-hours where mistakes cost the company tens of thousands of dollars an hour when I could be compensated the same for working on a team that has no real time restriction and mistakes have a small area of effect? It makes no sense. The only time anyone really pays attention to us is when something breaks. I get the idea that some members of management seem to have a lot of unrealistic expectations which we have really brought on ourselves by being so lucky in the past. They have no real understanding of the technical complexity of what it takes to guarantee 24/7 availability and yet they want to demand it on a whim.
I don’t think any of this is unique to where I work, either. It’s a larger problem with the way most business view their IT department. Technology is expensive, and people that are able and willing to manage it really well are hard to find. A lot of businesses see how much money they spend on IT and feel they’re entitled to demand just about anything. I can understand why they would feel that way. IT costs are ridiculous, but it still doesn’t mean anyone can act like they own the IT employees. We try to have lives outside of work just like anyone else.
Lucky for me, I don’t have to deal with the worst problems much longer. Oddly enough, according to HR, today is actually my first official day off the SAN team’s roster and onto the General Systems team where I will be doing application support. As part of the college grad hire program, you get moved around after 2 years. My physical move happens later this month where I’ll be going back to the main building. While I’m very excited about the move, I feel bad for those I’m leaving. They’ll have to keep dealing with stupid stuff like this.
Of course, maybe I’m wrong about how easy the applications team have it. I’ll find out soon enough.
My Trip to Canada September 30, 2008
Since my company has offices all around the world, our IT department regularly sends people to travel to those offices and perform security audits to make sure they aren’t doing anything stupid. Normally there is a specific team of people that handles all these trips, but this year they began taking an analyst along for the ride, part of their initiative to try and make all of us ‘think globally.’
Back in March, I found out that I was chosen to go on one of these trips. I was to go to Brazil and Chile in May. I was very excited about it at the time. But then weeks passed with no word on an actual date, which turned into months until I gave up on it ever actually happening. At the end of August, I finally got an e-mail out of nowhere saying a date was set for the trip, which turned out to be just over a month from that day. One month may seem like enough time to plan, but if you’ve ever taken an international business trip to a place like Brazil, you know that you can’t just flash your passport off the plane and start running around the place. You need a work visa, and that process is horribly tedious. A letter from the consulate in that country must be written for you requesting your visit. It must state your purpose in the most vague way possible; anything too specific, especially if it involves technology, will probably be rejected. Cramming that process into a month is pretty difficult, especially if doing it for the first time, which I would be doing along with the guy going with me who has done all the other audits.
Of course, none of that really mattered to me, because I had already booked a trip that week in October to visit my good friend, Aaron and crash Blizzcon. It just had to be that week, out of all the others that I had nothing going on. I e-mailed back asking if there was any possible way to move the trip around a week or two, but there were already other trips planned around it, so it wasn’t possible. Instead, I was asked if I wanted to go to Toronto, Canada instead. I replied that I would love to, and the Canada trip was set for September 24. Toronto would definitely be more low key and shorter than Brazil/Chile, but it also meant that I wouldn’t have to be driven around in an armored car or get any shots. I booked my flight and reserved my room.
Last Wednesday morning I left to go to work. I had everything packed and ready with me. It’s such a strange feeling knowing the next day you’ll be waking up somewhere you’ve never been before. Susan dropped me off at Matt’s (carpool buddy), who drove us to the office. I finished up a couple things and e-mailed my team letting them know I would be out the rest of the week along with a list of tasks I wasn’t able to get to that would probably need to be taken care of while I was gone. Matt then took me to the airport.
As we pulled up to the departing flights drop-off, my phone rang. It was Phil, my travel companion, and he had a bit of a problem. He was still going to Brazil in two weeks. After working for several weeks on the work visa process, he had finally reached the point where he could send off all his paperwork, so the day before the Canada trip, he sent it all off to Brazil. Interestingly enough, one of the items he mailed was his US passport.
For those new to geography, Canada is not the same as the United States. It is its own separate country, and you need a passport to go there. Phil told me to wait in the airport lobby without checking in while he called corporate travel. I wandered around the lobby for a few minutes and found the Air Canada check-in desk and took a seat nearby. About 10 minutes later, Phil called again. The trip would have to be rescheduled because there was no way he was leaving the country without a passport.
I called Matt to come pick me back up. Back at work the rest of the day, I had several conversations about the shortest Canada trip ever. I got to see the Nashville airport. I explored the lobby and saw where people go to claim their bags. I even caught a glimpse of where security checks you before you go on to the gates where the actual planes that fly to Canada are (or so I’ve heard). Next time, maybe I’ll get to see for myself.
It was very strange working and going home like normal when I was still in the mindset that I would be gone for the next few days. I called to cancel the flight and hotel room. Some of the cost will be credited towards the rescheduled trip, but I’m sure my company is eating some nice cancellation charges. We’re supposed to try this trip again at the end of October. After all this, I think I’ll believe it when I see it.
Gob and Lucy September 29, 2008
Brian and I are now proud owners of two furry family members. Most of you have had the pleasure of becoming acquainted with them by now as we are a little overdue in posting about them, but better late than never! Meet Gob and Lucy:
All Arrested Development fans out there should recognize their names. They are named after two of our favorite characters from a show that has provided us countless hours of Friday night entertainment in our first couple of years of marriage. Meet the real Gob (prounounced biblically like “Job”) and Lucy (short for Lucille):
We adopted them 2 weeks ago from a local shelter. We decided to get 2 kittens so that they would have a buddy, since we both work full time.
They certainly enjoy one another! We’ve only had them a couple of weeks and we already have tons of cute pictures and video to share, so check back soon for more cuteness!