Superbowl XXXII

On this day five years ago I was not at Superbowl XXXII. Instead, I was freezing my tail off in Nagano, Japan. 24 hours earlier (maybe somewhat more due to time-zone shifting) I had been in San Diego, CA, with full media access, supporting Silly me, I'd even packed my passport.

See, in the Fall of 1997, someone in IBM got pitched the idea of sponsoring the web site, and signed on. Now, initially this was just an advertising thing, however someone quickly pointed out that if IBM were sponsoring the site, everyone would assume that IBM was hosting the site, and wouldn't it be embarrassing if IBM didn't host the site?

Unfortunately, a number of people agreed and decided that IBM must design and host the Superbowl XXXII web site. My life took a decidedly negative turn.

See, IBM was also prepping to host the 1998 Nagano Olympic Games web site (then The core of IBM's brain trust in event hosting was pretty much pegged to setting up that site. And, in the spare time we had, also supporting the 1998 Australian Open web site.

Oh, and my day job was allegedly running But, no fear, IBM's Global Services - Network Services group jumped into the fray and say absolutely, postively, no problem hosting They had a whole new infrastructure set up just for these sorts of things, not the fancy, expensive RS/6000 SP systems we were using for and the Olympic Games web sites. Something new, something better.


On Friday the 23rd an IBM executive called up my manager and demanded that I be contributed to the Olympic Games effort, by working in an operations center in Schaumburg, IL. I had been a bit ambivalent about this, the Olympics can be fun, but I already had a couple fulltime jobs. A decision was made. (IBMese for "the decision has been made and you shall not challenge it") and I was to stop working on the Superbowl (heh, I was just superfluous anyway) and get to work on the Olympics. As part of this I agreed to travel to Nagano to meet up with my counterparts and get back up to speed on how the site was set up before travelling to Schaumburg, IL the following weekend to set up the US operations center. In theory, it would have been a good idea to pick up some winter clothing in San Diego. In practice, I didn't have that much time, and it was San Diego. Winter clothing is...well, not exactly easily found there.

Saturday the 25th, I left for the airport, and Alister left for the media center, to work on I flew halfway across the planet, got lost in the Ueno train station looking for the Shinkansen, and eventually made my way to the operations center for in the early evening of the 25th (Japan time).

Alister was still at work.

Apparently, while I was in transit, the "something new, something better" infrastructure had melted down. In technical terms, someone deleted root.cell from the AFS filserver. In practical terms, someone deleted the web site from the infrastructure. In reality, the site was all over the floor less than 24 hours before the game.

Meanwhile, in Nagano, Cam and Paul were busy not only working through infrastructure issues for the Olympic Games, but they were also busy rescuing both the Australian Open and Superbowl XXXII web sites.

Being totally wiped out, I left the ops center to check into my hotel room. AMEX had called ahead and had no problem getting a room, which was somewhat odd since Nagano was supposed to be sold out. Of course they had no problem getting a room for February 25th. Unfortunately for me, it was January 25th. Eventually I did get a room (at the Metropolitan I believe for one night, and then the Holiday Inn for the remaining two nights) and crashed.

Monday the 26th I got up and went over to the operations center where Cam and Paul were busy re-rescuing This time, something else "new and better" broke, fifteen minutes before kick off. So, while the primary team tried to get the site up and running in Schaumburg again, Cam and Paul had copied the entire site over to the Olympic infrastructure and were running it there. However, the Schaumburg team didn't want to turn over control of the site completely and were shunting only a small percentage of traffic over to the Olympic systems (which were built for far more traffic and were basically idling).

My contribution was pretty much to stay out of the way. I'd warned that we shouldn't try hosting the site and was overruled, and when I objected to certain aspects of the technical design of the site I was shot down as well.

Instead I focussed on coming up to speed on the design and operations of the naganoplex (as it was nicknamed) and the troubles that can ensue when you sequentially order two meals for yourself at the Nagano Gondo McDonald's (But sir, you have already eaten one meal. Why a second?).

A little over 48 hours after arriving I was back on the shinkansen to Tokyo-Ueno (nb: make sure that you get on the correct car next time as trying to walk forward through a bullet train as it accelerates, with luggage in tow, is not easy) and then on back home to New York City. I spent about two days in New York and then flew yet again, this time to Chicago, IL to the very offices where the site had melted down days before. Over the next four weeks I helped run during the US day time, and, well, was bored out of my mind since the damn thing just worked. The machines were idle for the most part, even at the peak rates (they were running the Lotus Go Webserver which wasn't too bad for certain tasks if you knew how to tune it correctly).

As far as I know, IBM has not hosted since.

«Ideology and Technology: Mode of Information as Critical Theory | Main |Your Single Point of Failure is your Best Employee »


Enter your email address:

Delivered by FeedBurner