How We Handled 1,400 Simultaneous Connections on Hotel Opening Day

Welcome furry fans!

We're glad you stopped by. Go ahead and register for a free account to get the benefits of being a member, including:
  • Access to all of our posts and comments
  • Your own profile including an avatar, buddy lists, and other social networking features
  • The ability to participate in a community of over 9,000 furry fans!
Creating an account is easy. Register now!
13 replies [Last post]
Giza's picture
This user is a Board MemberThis user is a Staff Member
Location: Ardmore, PA
"Webmaster Leopard"
Posts: 2859

FUR_0306

The Challenge

When hotel reservations open, that is the single busiest time of the year for Anthrocon's webserver. In fact, it even caused us performance problems last year. That was not so good.

So this year, I decided to try something different. Instead of leaving the regular website up and running, which involves using Drupal, I instead decided to replace the entire page with a relatively static "countdown" page, which displayed a countdown timer and automatically started displaying the hotel link at 11 AM on the opening day.

First, some stats for the Anthrocon website:

  • Peak bandwidth: 1.6 Megabits/sec
  • Peak connections: 1,400 concurrent connections

And now some status for Passkey, who handled most of the traffic:

  • Peak bandwidth: 190 Megabits/sec
  • Peak connections: 4,000 concurrent connections

Lightening the Load on the Webserver

With the kind of traffic mentioned above, I figured that the only way to keep the website from falling over again was to get Drupal out of the picture, at least for a few hours. Drupal is a fine Content Management System (CMS), but it's pretty heavy on CPU and database usage. That means it won't scale unless we throw tons of additional hardware at it. And it's not cost efficient to provision additional hardware that is used only once a year.

So the first thing I did was to write some PHP scripts that would serve up a series of static pages on the morning of February 1st. If the time was before 11 AM, a Javascript-based countdown timer would be displayed. If the time was greater than 11 AM on February 1st, a page with the link to hotel reservations would be displayed instead. Here are the screenshots of the pages in question:

But there was more to it than that. I needed to get as much traffic as possible away from our website. The next thing I did was to relocate our assets (image files, CSS files, and Javascript) onto Amazon's S3 service. At $0.12/Gigabyte for data transfer, it was fairly cheap. And Amazon had the infrastructure to handle lots of requests. Google was also a help here, as they host copies of jQuery that are available for public use.

Furthermore, I decided that to keep from hammering the CPU too hard, I would keep from accessing MySQL at all. That was easy enough to do, as I built the PHP scripts for serving up the read-only version of the website from the ground up.

Open Source

As a side note, I decided that I would contribute my code to the community after we were done with it, so I open-sourced my code and place it on GitHub: https://github.com/dmuth/anthrocon-hotel-countdown

Yes, we've open-sourced other projects over the years. The full list can be found here.

Social Media

We decided to use social media much more extensively this year. While last year it was an afterthought, this year it was a key part of our strategy to announce the hotel reservation link. We wrote up Tweets, Facebook posts, and Google Plus posts ahead of time. Then we announced those channels well in advance, as well as on our "countdown page", so that even if the website went down, we could get the word out, and people would know where to look.

When it came to be 11 AM on Friday morning, it was a simply copy and paste job to make the Tweets and posts with the hotel information. That helped divert even more traffic from our webserver.

Implementation

At 10 AM on Friday morning, I put the website into read-only mode. Since all pages in Drupal are served up through a single PHP script, switching over the website was a simple matter of running the UNIX command:

rm index.php && ln -s hotel-countdown/index.php .

The Results

The results were better than expected! Once the website was put into read-only mode, CPU and bandwidth usage dropped to near zero. In fact, even as the number of users visiting the site climbed both of those levels remained astonishingly low, up through and including 11 AM, the "zero hour".

At 11:15 AM traffic levels were on their way down, at which point I restored Drupal with the following UNIX command:

rm index.php && ln -s main.php index.php

Total time spent in read-only mode was 75 minutes.

Here are some graphs that better show the bandwidth, CPU usage, and number of concurrent connections:

Opening Anthrocon 2013 Hotel Reservations Opening Anthrocon 2013 Hotel Reservations Opening Anthrocon 2013 Hotel Reservations

Conclusions

Throughout the entire day, the website remained operational, with effectively 100% uptime. We received numerous compliments from our members on how smooth everything went, on both our end and Passkey's end. Here are some of the things that our members had to say about Passkey in our survey:

  • "Smooth, quick and surprisingly pain-free"
  • "Everything was so simple and streamlined"
  • "Smooth, intuitive, and zero lag"
  • "Smoothest experience I have ever had reserving a hotel room for Anthrocon"
  • "It was super quick and dead easy"

I think it is safe to say that using the Passkey system was an Epic Win, and that we can continue with this approach in future years.

4.32
Average: 4.3 (25 votes)
Your rating: None

Pennsylvania Furry? Be sure to check out http://www.pa-furry.org/

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
vinc's picture
Location: Newport, Michigan
"Wandering Dreamer"
Posts: 8

Look's like everything is running by....smoothly, im glad to hear it. 

5
Average: 5 (1 vote)
Your rating: None

●▬▬▬▬▬▬▬▬ஜ۩۞۩ஜ▬▬▬▬▬▬▬▬●

barkerjr's picture
Location: Hartford, CT, USA
"Purple Dragon in the Headless Zone"
Posts: 29

Congrats on a smooth execution Dance

5
Average: 5 (1 vote)
Your rating: None
Unclekage's picture
This user is a Board MemberThis user is a Staff Member
Location: RTP, North Carolina
Posts: 4346

Execution was a consideration if things had not run smoothly.

5
Average: 5 (3 votes)
Your rating: None
Remley Bishop's picture
Location: Waltham, MA, USA
Posts: 50

So you are saying that a Stable Execution, Stayed the Execution?  =^_^=

5
Average: 5 (1 vote)
Your rating: None
Unclekage's picture
This user is a Board MemberThis user is a Staff Member
Location: RTP, North Carolina
Posts: 4346

Quite! 

0
No votes yet
Your rating: None
Skyfox's picture
Location: Michigan, US
"I'm a fox, I'm a furry, and I'm mighty proud of that."
Posts: 206

Giza man, you and the staff rock! Cool

5
Average: 5 (2 votes)
Your rating: None

"Furries are everywhere, furries are everybody."--Me

BlixFox's picture
Location: Boonville New York
"Just your friendly neighborhood fursuit maker"
Posts: 1

hey Kage im gunna hi five ya if i see you at AC this year Sticking out tongue thanks for maintaining agreat Con!

0
No votes yet
Your rating: None

I was that person whp grew up wanting a fursuit after seeing listings for thousands of dollars i did my research and decided to build my own for way less than half the price, so i started construction for people with a risk and now have developed into

skunix's picture
Posts: 4

"We wrote up Tweets, Facebook posts, and Google Plus posts ahead of time."

Except you forgot email -_-

1
Average: 1 (2 votes)
Your rating: None
Unclekage's picture
This user is a Board MemberThis user is a Staff Member
Location: RTP, North Carolina
Posts: 4346

I hope you didn't expect us to email every Furry in the world to let them know...

0
No votes yet
Your rating: None
skunix's picture
Posts: 4

No but those of us who have attended before are on a mailing list correct?

I'm not trying to rain on your parade or anything, I think the new system is fantasitc and you handled it well.

I'm just saying an email telling previous antendees that you can't simply call up the hotel this year would have been nice.

1
Average: 1 (2 votes)
Your rating: None
Chiaroscuro's picture
This user is a Board MemberThis user is a Staff Member
Location: Norwich, CT
"Meef! Saute! Deal!"
Posts: 2794

Actually no, we don't place any prior attendees on a mailing list. That.. is an interesting idea. Mostly, we use the website, twitter, and massive word-of-mouth to inform attendees of significant changes, news, etc.

0
No votes yet
Your rating: None
Unclekage's picture
This user is a Board MemberThis user is a Staff Member
Location: RTP, North Carolina
Posts: 4346

We do not like the idea of sending mass email out to our attendees.  We already have a problem with Anthrocon email winding up in people's spam filters; the last thing we need is to get flagged as a bulk emailer.

0
No votes yet
Your rating: None
charlieg's picture
This user is a Contributor
Location: Alexandria, VA
"If only stupidity were painful..."
Posts: 1999

And that has already happened when Anthrocon sent out the e-mails to Supersponsors to give them the early registration information... and that was just over 200 e-mails.

What do you think will happen if Anthrocon tries to send out 4,000-5,000 e-mails?

0
No votes yet
Your rating: None

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.