I was thinking last weekend back to Easter 1997. It was an early Easter that year, on March 30th. I had spent the previous week in Raleigh, NC attending the "spring" Get Connected Technical Interchange (GetCoTI), an IBM internal conference. Due to the perennial budget crises within IBM, it turned out to be the GetCoTI for 1997 when its fall counterpart was cancelled.
Easter 1997 is the last Easter I remember as being a "good" family holiday.
Easter was, oddly, the big family holiday on my father's side of the family. We'd all get together at my grandparent's farm in Denham, IN and spend the day together. Over the years the activities changed, when we were younger my brother and I flew kites, then we progressed on to rockets. My father would hide "Easter eggs" (plastic eggs filled with jelly beans or other candy) and write semi-poetical rhymes as clues to find them. The biggest problem was not deciphering the rhymes, but deciphering my father's handwriting.
I wasn't planning on showing up, I had been in Raleigh all week, and my professional life at the time was rather chaotic. Reviewing my notebook from that period, it seems that I was in San Jose for a week earlier in March and spent a week in Austin in mid-April. At the last minute though, the night before, I called the travel people and changed my reservation to connect in Chicago, flying from RDU to ORD on the first flight in the morning and departing for LGA on the last flight that night. That gave me nearly twelve hours to spend in Chicago, enough time for the drive out and back to Denham. The downside was that my bags spent the day taking in the sights of the O'Hare baggage system.
I don't remember much about that day actually, I just remember enjoying it, being a bright spot in what was otherwise an incredibly stressful period at work. I got to see my grandfather, quite possibly for the last time though I don't remember now. He passed away a few months later (coincidentally I was again in Chicago for Internet World '97, but I didn't get to see him before he died). He had recovered from cancer, or so we all thought, and was up and around and quite spry. About a month later the doctors called and, this is my interpretation, said "oops" and told him they'd missed a spot, and, well, it was no longer a spot but a collection of spots. That sort of took the wind out of him.
But I enjoyed that day, I doubt there was kite flying or rocketry (or even target shooting, a former pastime of my grandfather).
1997 was a weird year for me.
In 1996 our office was highly encouraged to leave Armonk. When we did move out, the Armonk real estate folks demonstrated their love for our group by turning over our phone extensions to a new group the next immediate business day. I still wonder how whomever inherited 251-7516 felt on hearing he had inherited my phone number (on a good day I managed to fill the Phonemail "box" in the hour it took to drive from Poughkeepsie to Armonk).
We moved to 55 Broad Street, in lower Manhattan. I was so exhausted the week of the move that I fell asleep backstage during Lou Gerstner's "Network Computing" speech at the 1996 Fall Internet World at Javit's.
We spent the first five months at 55 Broad in a weird, bifurcated setup split between office suites on opposite corners of the 13th floor. Carol and Alex and their teams were in one suite, my www.ibm.com and w3.ibm.com teams held court in the back. I had a nifty little cubbyhole of an office which afforded me the ability to close the door and sleep.
I took advantage of that several times as I was attempting to commute from Poughkeepsie (well, New Hamburg) to Lower Manhattan daily. When I was driving to Armonk from Poughkeepsie I regularly stayed at IBM's luxurious management development center, or at the Greenwich Harbour Inn. It was somehow quasi-covered by my working arrangement.
But when the office moved to Manhattan I lost that "perq", and quickly moved up a plan to move to New York.
I ended up moving to Battery Park City (and as a result have far too many pictures of the south side of the World Trade Center ).
So, we had moved, had these temporary offices, and were trying to keep up the management of IBM's web presence. IBM real estate had sort of disregarded some of our requirements (well, most) for temporary space. As a result we had to use network connectivity from a reseller in the building. I vaguely recall sending Jean around to find out who could give us connectivity the day we moved in, since Advantis (IBM/Sears' networking business, which evolved to IBM Global Network before being discarded, er, sold to AT&T in 2000) …ahem, since Advantis quoted us a 90 window for connectivity.
We basically fell off the IBM map for awhile, which I wish I'd enjoyed more. Our tie-line numbers rang up at 33 Maiden Lane, not in our actual offices, so we had to check voicemail regularly. For whatever reason many IBM systems couldn't call our tie-lines, and no one wanted to actually call "long distance" so our call volume dropped dramatically.
Since we were using an unapproved ISP for our network connectivity, we could not access the IBM internal network except over dialup (this made life "fun" for the Intranet team).
But we worked this way through much of the winter of 1997 with few issues. Well, few that I can recall. In addition to running www.ibm.com, w3.ibm.com, and serving as the daily target for 300,000+ disaffected IBM employees who absolutely knew what we should be doing on the Internet; in addition we were planning a little event known as "Kasparov vs. Deep Blue: The Rematch" [Note: IBM maintains a pseudo-archive of the site at http://www.research.ibm.com/deepblue/home/html/b.shtml. I think the Internet Archive version is more true to what we actually served].
This soiree, this fete, this web event, well, looking back I'm surprised we did as well as we did, and I know there are many people who thought we didn't do well at all. I'm not talking about Deep Blue, which had a pretty good week and a half for a supercomputer; I'm talking about our editorial team which managed to pull together a wicked web site, and our technology team which tried to use every known web server in creation to keep the web site from falling over.
It is hard to explain this today, when my personal site has better technology and processing power than www.ibm.com did for its first two years, for under $500 a year.
We have to go back a bit, to June 1995. Lost amongst the headlines of the Lotus Development Corporation takeover was IBM Software Group's announcement of a range of Internet enabled software including a "web server". I, webmaster for www.ibm.com, first learned of this "web server" software about a week before the announcement and GA when someone called to complain that www.ibm.com wouldn't work on the IBM Internet Connection Server. Turns out IBM had produced a web server based off the old CERN server which, basically, did not work with CGI scripts. Which is how most of the applications ran on www.ibm.com at the time (pre-PHP, pre-Java, pre-Apache; this was NCSA 1.3 with a few special additions by yours truly).
Once ICS was announced we were expected to switch www.ibm.com over to it, well, immediately. The only problem was: it didn't work. The only realistic way to extend it was to write services, comparable to Apache modules. These services had to be perfect since they could bring the server down if they failed, they had to be reentrant because this was a threaded server. And in order for it all to run well, it pre-req'd IBM AIX 4.1.
At the time, IBM's hosting arm, which we were required to use, only supported AIX 3.2.something.
We committed to migrate to ICS (a/k/a Lotus Go Webserver, Domino Go Webserver, and eventually No-Go Webserver) when our hosting "partner" supported it, and when it could support the traffic (100k+ hits per day!) and applications we were running.
Let's return now to March 1997. We're working out of the temporary offices at 55 Broad Street, New York. And we're still running NCSA 1.3 (I think by this time I'd changed the server identifier to "IBM Planetwide", which I'd also used for the web crawler and a couple of other places on ibm.com). By this point the "Go" webserver had been on the market for 20 months, and ibm.com still didn't use it. Our relationship with the Software Group was a little tense (we had already been burned by turning over the search engine to a number of parties within IBM Software).
By March 1997 we thought we were set to shift off NCSA 1.3 to "Go". We had cleaned up the CGI scripts, we had written a couple of services, we had migrated to AIX 4.x, we'd done a lot of work. We ran a number of test runs with what we thought was a representative load and all seemed ok.
We made the switch on a Saturday, around the middle of March, to IBM's premier web server, I'm fairly certain by then it was "Lotus Go Webserver".
About four chaotic, error-and-outage filled hours later we switched back. It just could not handle CGI scripts. At all. What we learned then, and would painfully get in-depth education on a few weeks later, is that multithreaded applications absolutely detest forking. A fork() call to a multithreaded application (at least in 1997) was like an insult to its mother, a slander on its brute force potential power. Didn't want to do it, wasn't going to do it, only if you waited patiently for each thread to suspend would it eventually consider just maybe, maybe forking to run a CGI.
Now, this was a painful educational experience, moreso because we had committed to using Lotus Go Webserver as the webserver for the web event of the year: Kasparov vs. Deep Blue: The Rematch. However, we had carefully architected www.chess.ibm.com to have as few CGI scripts as possible. We had outsourced the chat portion of the site to Howard Rheingold's Electric Minds. The remaining CGI scripts we'd redirect off to an Apache or NCSA server. All would be well.
Edward, what a silly optimist you were back then.
About three weeks later we were chugging along. I'd returned from my little Easter sojourn to Indiana. Our new offices were progressing, we'd be able to move in the week before Deep Blue started. ibm.com was running on a few nodes of our Columbus SP complex, the rest were being used by our friend and colleague Paul Reed to run the Master's web site.
Last weekend, when I started thinking about this piece, there were a number of events that occurred to open the memory floodgates. It was Easter. I was watching "Deep Impact", which isn't a very cheery movie, but has a dialogue between Tea Leoni's character and the character's father about "one perfect day". And I was watching Tiger play (and eventually lose) the 2007 Masters Tournament.
Back in 1997, Paul was unhappy about Tiger's presence at the Masters. See, in 1996 the Masters web site had been pretty low key. A burst of traffic, but not insane. Not like that year's ACM Chess Challenge (the first Deep Blue site), or the Olympic Games, or a tennis match. No, golf was more civilized, people were not as interested.
Until Tiger Woods played and won the 1997 Masters Championship.
Unfortunately, while Paul was getting hammered by Tiger's fans hitting the web site, I was apparently in a meeting in Austin, TX (the ICC meeting) and have no interesting notes to add (my notebook is eerily silent about the '97 Masters). I think that he managed to use every webserver we had compiled on the ibm.com complex and don't remember if he ever found one that worked well.
Still, I was confident that we had planned out www.chess.ibm.com well, that Lotus Go webserver would work well, and our operations room would be serene.
Silly, silly not-quite-30 year old man.
The next few weeks were filled with testing, final details, a move from the 13th to 27th floors to our new offices.
I really thought we would do fine, perhaps a few rough moments, but I did not expect a disaster.
The 1996 match had been a disaster.
At that time IBM had no intention of doing "event" web sites like this, other than the sports web sites.
So the 1996 ACM Chess Challenge web site, aka www.chess.ibm.park.org, was outsourced to a Boston based web hosting company, who stuck it on a shared hosting server.
No one is going to watch a chess game online, right?
The online offering was our display for the 1996 Internet World Expo (has there been one since?)
Right, well 1996 was a disaster.
I am not going to go into detail about what happened for the rematch, suffice it to say www.chess.ibm.com melted down promptly at 3:00 p.m. on the first day of the match. At 2:59 it was running fine. At 3:00 the entire world rushed in and well, what can I say, I fried an RS/6000 43P which was tailing logs off all 24 servers.
Over the course of the next eight days we too tried using various configurations, Apache, NCSA, to no avail. Once the match started we were hosed. We had various distractions like the attempt to port the site to the idle "womplex" hosting complex. We learned far more than we wanted to know about the complexities of multihomed systems running AFS, AIX 4.x, and multithreaded webservers. AFS was not threadsafe, nor was the domain name resolver library. DNS should not have been an issue but for some odd routine which, even though lookups were turned off, was doing lookups anyway. On each hit.
At the last possible minute before the match started, we were thrown a curveball: IBM had agreed to partner with Garry Kasparov in Club Kasparov, including hosting his web site.
That apparently meant we had to host the clubkasparov.com web site.
So we did a rush-port of a web site we knew nothing about to our hosting complex, while in the last stages of preparing for the onslaught.
On the plus side, we had our first international mirror (www.uk.chess.ibm.com). We probably pioneered the idea of a stripped down home page, when we set www.ibm.com to use a very stripped down page, offering people to go either to the chess web site, or ibm.com proper.
We won the Cool Site of the Year for Live-Online Event award for 1997.
I think we may have had some help from other IBM employees, but not much (I recall getting "escalated" over even mentioning the contest on the old WWWIBM FORUM).
Most of all, this was the last "event" web site we agreed to do. Well, almost.
But it was ludicrous for us to think we could pull off our day jobs running IBM's web presence, and pull together and run www.chess.ibm.com, and do both with grace and aplomb.
In the aftermath of Deep Blue's "victory", Kasparov made various accusations and insinuations against IBM and its employees.
That wasn't well received, and just as quickly as we were told to host the Club Kasparov web site, we were told to shut it down and remove it from the IBM servers.
I was so fried afterwards that on May 12th I declared I was going on vacation and that I'd return some time in June. I spent the first three days sitting in my apartment, unpacking the boxes which had remained packed since I'd moved in in February. On the fourth day I realized that I would never actually go anywhere on vacation unless I physically left the apartment and promptly bought a ticket to Florida (where I'd spend the next week in quiet isolation. There was a protocol to follow to reach me involving calling my parents, reciting the passphrase, and only to be done if it was absolutely necessary. I think Carol, who is reading this now, managed to hold out for almost a week before someone came up with a "The Chairman demands…" request).
This would turn out to be the last vacation of any real consequence for more than a year (somehow I ended up in Florida in the summer for my "break" in 1998 as well).
The disaster we encountered with using IBM software for Deep Blue II had some beneficial outcomes, the Lotus Go product underwent extensive remodeling and performed flawlessly for the 1998 Nagano Olympic Games web site. Later in 1998 we finally moved www.ibm.com to the server, now called Domino Go Web Server. And I think two weeks later IBM killed the product, announcing its support for Apache, and putting me in the bizarre position of being criticized on an IBM conference call for being "against" open software (having run ibm.com on "open software" for most of the time from 1994 through 1997).
1997 was a weird year, full of some professional highs, many professional lows. My grandfather died in July 1997. I continued to be on the receiving end of escalation after escalation (I no longer had the ability to hot-transfer people to the Chairman's office when they threatened to escalate me to Lou). My father had a silent heart attack (well, we weren't so sure afterwards that it was so silent) in the fall followed by a massive heart attack in December. He underwent a quadruple bypass and survived, in a way, but the bypass had the sick side effect of aggravating circulation issues from diabetes, leading to a two year cycle of hospitalizations, amputations, and recovery. He would die in May 2000.
When I talk to people about working at www.ibm.com, I look back on 1997 and 1998 as my "glory years". It's easier to do that now, ten years later, but we did get on a roll. We had a great office at 55 Broad, for much of that period we had great management support. The "edplex" as the ibm.com complex got nicknamed served well, even as it was clearly aging rapidly. For 66Mhz systems it didn't perform too badly most of the time.
But I also remember 1997 as the last good Easter with my extended family. I don't remember Easter 1998 or 1999. And Easter 2000 occurred two weeks before my father died. I did get to see him, en route amongst Sydney, New York, Lausanne and San Francisco. He didn't look well, but he hadn't looked well for the past year.
And now, now I don't really celebrate Easter. It never was much of a religious holy day for me (going to the 4 hour Easter Vigil mass the night before, so you can get an early start on the Tri-state to Denham, that 4 hour mass tends to diminish the religious fervor). And since my grandfather died we've (well, I, who admittedly is 800+ miles from the center of gravity for my extended family) we've stopped using Easter as a family get together.
And I find, mostly, that I'm not missing it and seem to be ok with that.
Coincidentally, our "foe" from 1997 is now a political activist in Russia: Garry Kasparov was arrested this past weekend at a protest in Moscow.
I wish him no harm and hope he succeeds in his campaign against the rise of totalitarianism in Russia.
In writing this I'm amazed that these sites from 1996 and 1997 are still mostly accessible.
It's too bad that IBM can't spare a few cycles to host the sites themselves in the way they were presented then.
The Internet Archive is nice, but slow, and seems to have problems fetching and storing images (I cannot tell if they
e.p.c. posted this at 11:00 GMT on 16-Apr-2007 from Brooklyn, NY.
, Comments