Brian and Susan

Just another weblog

Production Outage October 1, 2008

Filed under: Work — Brian @ 2:04 pm
Tags: , , ,
I took the failboat out for a joyride.

(or how I rode the failboat out for a joyride)

Last night I took down about 6 servers for work.  Important ones, too.  It was supposed to be just a small fix to the zoning on our main SAN switches for something unusual that happened yesterday during the day (human error caused by a new team member who I also didn’t stop at the time).  The original problem wasn’t a huge deal, but it created a small mess than needed to be cleaned up.

Apparently, I committed the fix wrong while cleaning it up later that night.  It’s really easy to do.  I’m kind of surprised it doesn’t happen more often.  There’s an A-side switch and a B-side switch (for redundancy), and if you accidentally activate changes on the B-side with the A-side selected, bad things happen.  And they did.  Every server lost connection with the B-side of our storage network.  Fortunately, I only made the mistake on one side, which means everything stayed connected on the A-side, so most servers could still see their storage but just had fewer “paths” to get to it.  Unfortunately, not all servers are configured correctly to handle losing paths (or worse, still aren’t even connected to both sides).  Those were the ones that went down.  Thank God it was only one side.  Accidentally bringing down all 200 physical servers and another 400 virtual servers would have been a total disaster, even after business hours.  I try not to think about it, but the fact that it’s that easy for me to do something like this is scary to say the least.

There’s a root cause meeting going on now.  No doubt the production control police will be grilling my team lead as to why there was no change ticket for this ‘change’ when it really wasn’t even supposed to be a change at all.  It hasn’t been a very good couple of months for infrastructure.  We’ve had a lot of stupid outages lately.  This does not help my team out at all.  The demand for only touching production devices on Saturdays will definitely rise.  Those meetings can be extremely annoying, too.  When something like this happens, you get to try and explain what happened to the production control team, people who generally have no clue what you’re talking about and seem to think that “if only you had created a ticket” or “if only this had been done on a Saturday” nothing would have happened.  The truth is that outages are going to happen eventually.  People make mistakes, and hardware fails.  Saying “we must have 24/7 availability” does not instantly make it so, especially when you don’t wish to pay for that kind of environment and also expect your employees to work terrible hours and pretend that it’s “just part of their job.”  It may be part of the job, but the way infrastructure employees work around here goes far beyond what I consider their core job, especially when compared to the rest of IT.  I have managed to dodge most of the after-hours work on this team due to most of my contributions being scripting; most others have not been so lucky.

Things like this make me question a future in the IT department, at least on a lower-level infrastructure team.  I’m not a nights and weekends kinda guy when it comes to my job, especially if I’m not getting paid overtime and/or a shift differential of some sort.  Back in the day, IT apparently used to get overtime here as well as reimbursement for being forced to travel between two data centers, but those days are long gone for most all companies now because they became “too expensive.”  Instead, we’re supposed to take “comp time” where if you work a Saturday, you take off the next Friday or something.  The main problem is that pretty much no one ever takes their full share of comp time, usually because they have too much crap going on during the day to just take off.  Another problem is that having some random day off during the week does not make up for losing a Saturday.  When I have Saturday off, I can hang out with my wife and other friends/family.  When I have Friday off, I sit at home by myself doing nothing while everyone else is at work.

It isn’t that I’m not paid well, it’s that I’m paid the same as other people on application teams who almost never have to wait until after-hours to actually do their work.  Those teams also deal with a much lower degree of personal risk.  If they make a mistake, at worst they probably just screw up their one application.  If I make a mistake, I can easily bring down the entire IT infrastructure of my business unit.  Managing millions of dollars of SAN equipment has been very satisfying.  It really is cool stuff.  But it touches almost every single server (hundreds and hundreds) in our environment.  Centralized storage makes things much more efficient, flexible, and generally more awesome.  It also means any outages to your SAN are absolutely devastating.  When servers lose their storage, they basically fall over and enter a coma which can only be remedied by the server teams very tediously logging into every single one and rebooting.  That’s assuming no data corruption, which takes a lot more work to fix.  It’s also assuming people remember how to bring up a server+application which might not have been rebooted in over a year.  Oh, and if an outage does happen during the day and some of the really important stuff goes down, God save you from the wrath of everyone else in IT because you just negatively impacted a very important part of their quarterly bonus.  People get angry when you’re the reason they get paid less.

I’m sure the idea is that everyone in IT is equal and no one should be treated differently.  It’s just not true.  Some jobs are harder or more important than others.  Having worked on one of if not the most critical team, I can say that it’s just not worth it if there isn’t some other incentive. Why would I want to work on a team that has to do most everything after-hours where mistakes cost the company tens of thousands of dollars an hour when I could be compensated the same for working on a team that has no real time restriction and mistakes have a small area of effect?  It makes no sense.  The only time anyone really pays attention to us is when something breaks.  I get the idea that some members of management seem to have a lot of unrealistic expectations which we have really brought on ourselves by being so lucky in the past.  They have no real understanding of the technical complexity of what it takes to guarantee 24/7 availability and yet they want to demand it on a whim.

I don’t think any of this is unique to where I work, either.  It’s a larger problem with the way most business view their IT department.  Technology is expensive, and people that are able and willing to manage it really well are hard to find.  A lot of businesses see how much money they spend on IT and feel they’re entitled to demand just about anything.  I can understand why they would feel that way.  IT costs are ridiculous, but it still doesn’t mean anyone can act like they own the IT employees.  We try to have lives outside of work just like anyone else.

Lucky for me, I don’t have to deal with the worst problems much longer.  Oddly enough, according to HR, today is actually my first official day off the SAN team’s roster and onto the General Systems team where I will be doing application support.  As part of the college grad hire program, you get moved around after 2 years.  My physical move happens later this month where I’ll be going back to the main building.  While I’m very excited about the move, I feel bad for those I’m leaving.  They’ll have to keep dealing with stupid stuff like this.

Of course, maybe I’m wrong about how easy the applications team have it.  I’ll find out soon enough.


My Trip to Canada September 30, 2008

Filed under: Work — Brian @ 2:07 pm
Tags: , , , , , , ,

Since my company has offices all around the world, our IT department regularly sends people to travel to those offices and perform security audits to make sure they aren’t doing anything stupid.  Normally there is a specific team of people that handles all these trips, but this year they began taking an analyst along for the ride, part of their initiative to try and make all of us ‘think globally.’

Back in March, I found out that I was chosen to go on one of these trips.  I was to go to Brazil and Chile in May.  I was very excited about it at the time.  But then weeks passed with no word on an actual date, which turned into months until I gave up on it ever actually happening.  At the end of August, I finally got an e-mail out of nowhere saying a date was set for the trip, which turned out to be just over a month from that day.  One month may seem like enough time to plan, but if you’ve ever taken an international business trip to a place like Brazil, you know that you can’t just flash your passport off the plane and start running around the place.  You need a work visa, and that process is horribly tedious.  A letter from the consulate in that country must be written for you requesting your visit.  It must state your purpose in the most vague way possible; anything too specific, especially if it involves technology, will probably be rejected.  Cramming that process into a month is pretty difficult, especially if doing it for the first time, which I would be doing along with the guy going with me who has done all the other audits.

Of course, none of that really mattered to me, because I had already booked a trip that week in October to visit my good friend, Aaron and crash Blizzcon.  It just had to be that week, out of all the others that I had nothing going on.  I e-mailed back asking if there was any possible way to move the trip around a week or two, but there were already other trips planned around it, so it wasn’t possible.  Instead, I was asked if I wanted to go to Toronto, Canada instead.  I replied that I would love to, and the Canada trip was set for September 24.  Toronto would definitely be more low key and shorter than Brazil/Chile, but it also meant that I wouldn’t have to be driven around in an armored car or get any shots.  I booked my flight and reserved my room.

Last Wednesday morning I left to go to work.  I had everything packed and ready with me.  It’s such a strange feeling knowing the next day you’ll be waking up somewhere you’ve never been before.  Susan dropped me off at Matt’s (carpool buddy), who drove us to the office.  I finished up a couple things and e-mailed my team letting them know I would be out the rest of the week along with a list of tasks I wasn’t able to get to that would probably need to be taken care of while I was gone.  Matt then took me to the airport.

As we pulled up to the departing flights drop-off, my phone rang.  It was Phil, my travel companion, and he had a bit of a problem.  He was still going to Brazil in two weeks.  After working for several weeks on the work visa process, he had finally reached the point where he could send off all his paperwork, so the day before the Canada trip, he sent it all off to Brazil.  Interestingly enough, one of the items he mailed was his US passport.

For those new to geography, Canada is not the same as the United States.  It is its own separate country, and you need a passport to go there.  Phil told me to wait in the airport lobby without checking in while he called corporate travel.  I wandered around the lobby for a few minutes and found the Air Canada check-in desk and took a seat nearby.  About 10 minutes later, Phil called again.  The trip would have to be rescheduled because there was no way he was leaving the country without a passport.

I called Matt to come pick me back up.  Back at work the rest of the day, I had several conversations about the shortest Canada trip ever.  I got to see the Nashville airport. I explored the lobby and saw where people go to claim their bags.  I even caught a glimpse of where security checks you before you go on to the gates where the actual planes that fly to Canada are (or so I’ve heard).  Next time, maybe I’ll get to see for myself.

It was very strange working and going home like normal when I was still in the mindset that I would be gone for the next few days.  I called to cancel the flight and hotel room.  Some of the cost will be credited towards the rescheduled trip, but I’m sure my company is eating some nice cancellation charges.  We’re supposed to try this trip again at the end of October.  After all this, I think I’ll believe it when I see it.


So…what exactly do you do? June 26, 2008

Filed under: Work — Brian @ 10:13 pm

I get this question a lot, and I’ve never really thought of a good short answer.  So here’s the long one.  By title, I’m an IT Analyst I.  It pretty much just means I mostly mess around with technology all day while also keeping up with the normal stuff of just working in an office – going to team meetings, writing up weekly reports, going to lunch, writing  yearly goals, etc.

More specifically, I work for Caterpillar Financial, a wholly-owned subsidiary of the extremely massive Fortune 100 equipment manufacturer Caterpillar, Inc., based in Peroria, IL.  To give an idea of what our main business is, let’s say your old family D9 is getting pretty outdated and your spouse has become embarrassed having to drive it around because all his or her friends have the newer models already.  You really need something you can comfortably sit the small kids in while you’re excavating your front yard.  So you go to a CAT dealer to buy a newer, more powerful D10:

D10.  Under $1 million!  What a steal!
D10. Under $1 million! What a steal!

Now, you’re probably going to want some financing to go with it.  You could get a loan from some other place like a bank, credit union, shady guy in an alley OR you could get a loan from CAT Financial.  Hopefully you choose the latter because of the superior quality and service you can get from being financed by the people who also happen to know the product the best.  (That option also helps pay my bills).  CAT Financial also does things like insure equipment (includes interesting things like semi-truck engines and boat engines), resell used equipment, recycle parts of old equipment, manage transporting machines around the world depending on dealer inventory.  Fun fact: the owner of the most pieces of CAT equipment is actually Caterpillar Financial.  Maintaining a tight control on these machines allows CAT to also keep a tight control on quality, which protects a very strong brand image.

As for me, I’m currently in the IT department on the Storage Support Team.  The computer you’re reading this on has a hard drive in it, probably with somewhere between 50-200GB of space.  If you don’t have a reference for what that means, you can probably fit about 40,000 average-sized MS Word documents in 1GB.  I deal with servers in my job.  Those servers need hard drives, too.  Rather than put hard drives in each one, we have giant electronic cabinets that hold hundreds and hundreds of hard drives.  The servers are connected to these central cabinets using fibre optic cables and get their storage from them.  We manage about 140TB of storage, which will be over 200 easily by next year.

Still figuring out how to fit one in my house...

I spend a lot of time doing things like: configuring space allocation for storage requests, configuring switches, writing scripts for tedious tasks, writing insane SQL queries to create cool reports, upgrading firmware, troubleshooting backup issues, and replacing bad parts.  Fortunately, I don’t have to do a lot of evening or weekend work like the server admins do.

By January I’m scheduled to be moving to another team – an applications team.  It will be much more software-oriented, probably doing more programming.  While I don’t like the idea of being a huge noob all over again, I’m excited to learn stuff like .NET.  One of the most significant things about moving is that I get to move back to the nice headquarters building downtown, so Susan and I should be able to do some carpooling.  My team was moved out into the data center building last August as part of a reorg thing.  I’m ready to be back in the main building.

Oh, I also occasionally do things like write blog entries instead of doing actual work.  Heh.


I Bring Tamale

Filed under: Work — Susan @ 4:30 am

I thought I would try to start to remember some funny things that have happened so far in my nursing career as I have told the stories but not written them down. Here’s one of my favorites from my last job:

Scenario: Me taking care of a Hispanic family with pretty decent English, in clinic for a stomach bug in their daughter.

Mom: “When I call they say you have lab.”
Me: “Yes, we have a lab, and we can perform certain lab tests if we need them and send out others we can’t complete here.”
Mom: “Ok. I think she sick because of tamale she ate yesterday. Could this be?”
Me: “It’s certainly possible that she could have gotten sick after eating a tamale, but…”
Mom: “Ok, I want lab test.”
Me: “Well, I don’t think there are any lab tests we need on your daughter today as long as she stays well hydrated.”
Mom: “No, not for her. (reaches into diaper bag and pulls out large square of aluminum foil) I bring tamale.”

This same family was very opposed to seeing an NP when I first started. I think they forgot that I was an NP, as at the end of this visit the mom said, “Oh, you getting good at this now! You fast! You almost doctor, right?” I had explained that I was an NP in the past, and this day I said, “no, I am a nurse practitioner…” mom: “yeah but you almost doctor right? You good now!” I thought for a second, and was like, “sure…I’m almost a doctor” meaning that we have similar job descriptions. It was much easier than explaining the role of an NP. One of the office workers stated that in Mexico there is no word for nurse practitioner, so it is a confusing term for many people not from this country. After that visit that family was fine seeing me!

Also of note, we did not draw labs on the tamale 🙂


Water Off a Duck’s Back

Filed under: Work — Susan @ 4:25 am

In clinic this week I encountered my first family at Vanderbilt that did not want to see me (because I’m an NP and not a doctor). This happened more than a few times in school and in my last job, but it still stings every time.

Today was the worst – I walked in and said hello to the family, and immediately the mom says, “We saw you last time and you gave us a lot of misinformation, and we don’t want to see you again.” Um, I’m 100% certain that their last visit was 6 months ago and I didn’t even work there at the time. I told them that they had not seen me before, and they said “well it was another NP and we didn’t like her either and we’d prefer not to speak to anyone but the doctor.” Also 100% sure that their last visit was with a resident, aka MD, not an NP. Either way they refused to see me.

I know not to take it personally, but it still gets me a little. I hate that they had a previous unsatisfactory visit, but I had nothing to do with it, and what they (and many others) don’t realize is that all good NPs are great at knowing what they don’t know – meaning I will always get the doctor if there is anything I am unsure of. I know I am biased, but I love the role of the NP. Most patients don’t mind seeing an NP at all, and many even prefer it! To each his own…I just still need to work on the whole “like water off a duck’s back” thing.


How Does Your Garden Grow?

Filed under: Play,Work — Susan @ 4:22 am

Brian and I now have this great yard and would love to have a garden; thing is, we know nothing about gardening. My great co-workers at MCC decided to jump-start our gardening hobby by giving me everything but soil as my going away gift. I couldn’t think of a more thoughtful gift – some people gave parts of plants from their own gardens or plants that are meaningful to them.

Due to the temporal nature of plants with limited space for roots, Brian and I dove right in and with the generous help of my family, added on to our existing flower bed in the front lawn. Here are a few photos from our garden party.

Original post 5-15-08


Status Post Recent Change

Filed under: Play,Work — Susan @ 4:09 am

May has been an eventful month thus far.

May 1: My last official day at Mercy. What an amazing last day! My co-workers were so encouraging and kind – they threw a surprise lunch fiesta with incredible authentic Mexican food, a very funny (and sweet!) group poem, and a gift I never expected…more on that in a later post.

May 3: Sister-in-law Stephanie graduates from MTSU. Hooray for the amazing teacher to be! She also found out that she will have a job at the school she’d hoped to work for. Great news! (update 6-25-08: turns out that Stephanie didn’t get the job she wanted due to a relative working at the school, so she is currently searching for something wonderful which I’m sure she will find!)

May 9: My graduation from Vanderbilt. Even though I finished school 9 months ago my graduation ceremony wasn’t until now so that the school of nursing would graduate with the rest of Vanderbilt University – a much larger group than I had thought! It was very Hogwarts-esque with lots of pomp and circumstance and funny hats. The ceremony was outdoors…the weather was perfect!

May 11: Happy Mother’s Day!

May 13: My 24th birthday!

In the past couple of days Brian has found out that he will move to the downtown building so we will be able to carpool to work starting in September…I am so looking forward to spending that time with one another. Half way through the month and look at all that has gone on…we are very happily busy.

Original post 5-15-08