How We Handled 1,400 Simultaneous Connections on Hotel Opening Day
When hotel reservations open, that is the single busiest time of the year for Anthrocon's webserver. In fact, it even caused us performance problems last year. That was not so good.
So this year, I decided to try something different. Instead of leaving the regular website up and running, which involves using Drupal, I instead decided to replace the entire page with a relatively static "countdown" page, which displayed a countdown timer and automatically started displaying the hotel link at 11 AM on the opening day.
First, some stats for the Anthrocon website:
- Peak bandwidth: 1.6 Megabits/sec
- Peak connections: 1,400 concurrent connections
And now some status for Passkey, who handled most of the traffic:
- Peak bandwidth: 190 Megabits/sec
- Peak connections: 4,000 concurrent connections
Lightening the Load on the Webserver
With the kind of traffic mentioned above, I figured that the only way to keep the website from falling over again was to get Drupal out of the picture, at least for a few hours. Drupal is a fine Content Management System (CMS), but it's pretty heavy on CPU and database usage. That means it won't scale unless we throw tons of additional hardware at it. And it's not cost efficient to provision additional hardware that is used only once a year.
Furthermore, I decided that to keep from hammering the CPU too hard, I would keep from accessing MySQL at all. That was easy enough to do, as I built the PHP scripts for serving up the read-only version of the website from the ground up.
As a side note, I decided that I would contribute my code to the community after we were done with it, so I open-sourced my code and place it on GitHub: https://github.com/dmuth/anthrocon-hotel-countdown
Yes, we've open-sourced other projects over the years. The full list can be found here.
We decided to use social media much more extensively this year. While last year it was an afterthought, this year it was a key part of our strategy to announce the hotel reservation link. We wrote up Tweets, Facebook posts, and Google Plus posts ahead of time. Then we announced those channels well in advance, as well as on our "countdown page", so that even if the website went down, we could get the word out, and people would know where to look.
When it came to be 11 AM on Friday morning, it was a simply copy and paste job to make the Tweets and posts with the hotel information. That helped divert even more traffic from our webserver.
At 10 AM on Friday morning, I put the website into read-only mode. Since all pages in Drupal are served up through a single PHP script, switching over the website was a simple matter of running the UNIX command:
rm index.php && ln -s hotel-countdown/index.php .
The results were better than expected! Once the website was put into read-only mode, CPU and bandwidth usage dropped to near zero. In fact, even as the number of users visiting the site climbed both of those levels remained astonishingly low, up through and including 11 AM, the "zero hour".
At 11:15 AM traffic levels were on their way down, at which point I restored Drupal with the following UNIX command:
rm index.php && ln -s main.php index.php
Total time spent in read-only mode was 75 minutes.
Here are some graphs that better show the bandwidth, CPU usage, and number of concurrent connections:
Throughout the entire day, the website remained operational, with effectively 100% uptime. We received numerous compliments from our members on how smooth everything went, on both our end and Passkey's end. Here are some of the things that our members had to say about Passkey in our survey:
- "Smooth, quick and surprisingly pain-free"
- "Everything was so simple and streamlined"
- "Smooth, intuitive, and zero lag"
- "Smoothest experience I have ever had reserving a hotel room for Anthrocon"
- "It was super quick and dead easy"
I think it is safe to say that using the Passkey system was an Epic Win, and that we can continue with this approach in future years.
Look's like everything is running by....smoothly, im glad to hear it.
Congrats on a smooth execution
Execution was a consideration if things had not run smoothly.
Giza man, you and the staff rock!
hey Kage im gunna hi five ya if i see you at AC this year thanks for maintaining agreat Con!
"We wrote up Tweets, Facebook posts, and Google Plus posts ahead of time."
Except you forgot email -_-
I hope you didn't expect us to email every Furry in the world to let them know...
No but those of us who have attended before are on a mailing list correct?
I'm not trying to rain on your parade or anything, I think the new system is fantasitc and you handled it well.
I'm just saying an email telling previous antendees that you can't simply call up the hotel this year would have been nice.
Actually no, we don't place any prior attendees on a mailing list. That.. is an interesting idea. Mostly, we use the website, twitter, and massive word-of-mouth to inform attendees of significant changes, news, etc.
We do not like the idea of sending mass email out to our attendees. We already have a problem with Anthrocon email winding up in people's spam filters; the last thing we need is to get flagged as a bulk emailer.
And that has already happened when Anthrocon sent out the e-mails to Supersponsors to give them the early registration information... and that was just over 200 e-mails.
What do you think will happen if Anthrocon tries to send out 4,000-5,000 e-mails?