Converting instance-store instances to EBS instances (AWS EC2)

February 27th, 2010 by Tony No comments »

A month or two ago, Amazon Web Services (AWS) announced that their EC2 instances will now be bootable from an elastic block store (EBS) volume.  This seems like a small change, but in fact has opened up a world of possibilities in the Elastic Computing Cloud (EC2).

EBS provides “block level storage volumes for use with Amazon EC2 instances.”  Keep in mind that prior to this, instances were limited to booting off of S3-backed Amazon Machine Images (AMI), which were not persistent images.  This meant 2 things:

  1. instance-store AMIs cannot be “stopped,” only rebooted or terminated
  2. rebooting an instance-store AMI reverted the instance back to the AMI defaults

Since EBS originally debuted as a high-performance attachable block device to a given instance, booting off of EBS AMIs has shown to be faster than the traditional instance-store boot as well.

Another benefit is that EC2 instances booted off of EBS volumes can be stopped, which effectively equates to shutting down a machine in the real world.  You are still responsible for the charges incurred while this instance is reserved, but all of the changes made to said instance will persist after you start it again.

The last major benefit of booting off the EBS volumes is that AWS has made it easy for you to create a new EBS AMI from a running EBS AMI.  In the Console, right clicking on an EBS instance will yield a new option, “Create Image (EBS AMI).”  This will basically shut down your instance, and proceed to generate a new EBS AMI from the contents on the disk of your instance.  This command seems to have a failure rate of ~40%, which can be a little frustrating.  I’ve found that if you put an instance into ’stopped’ state before creating the EBS AMI, the process has a higher chance of success, but will still take anywhere from 20-45 minutes.

The rest of this article will focus on converting an instance-store AMI into an EBS AMI.

In order to perform this conversion, you will need to have an instance-store AMI that is the base OS you’d like to run (for the purposes of this article I used alestic’s Debian 5.0 base image), and access to EC2 via CLI as well as the portal (it’s all do-able from the CLI, but some of the tasks are a LOT easier and quicker through the web console).  The stuff I did in the console will be suffixed with [console],  and the stuff from CLI will be prefixed with #.

1) Booting an instance-store AMI – I executed the following to get a list of the images that fit my criteria (32bit, Debian, base install):

# ./ec2-describe-images –region eu-west-1 –all | grep -i lenny-base | grep i386

IMAGE   ami-b13a6bf4    alestic-32-us-west-1/debian-5.0-lenny-base-20090804.manifest.xml
IMAGE   ami-b33a6bf6    alestic-32-us-west-1/debian-5.0-lenny-base-20091011.manifest.xml

Note:  ec2-describe-images outputs way more data than this, above is formatted for brevity.

Once you have the AMI (newer is better, generally speaking), boot an instance with this AMI:

# ./ec2-run –region eu-west-1 -k $keypair  ami-b33a6bf6

Note: $keypair in this case is the name of keypair used to SSH into the server

2) Customizing the EBS volume – After the instance is up and running, look to see which availability zone the instance is in.  If the region is eu-west-1, the availability zone is going to be either eu-west-1a, or eu-west-1b.  In either case, find out which availability zone your instance is in, and then create a 10gb EBS volume is the same zone [console].

Why 10gb?  10gb is the maximum size for an S3-backed AMI, which makes a 10gb volume the largest any instance-store AMI will be.  Obviously EBS AMIs can exist on larger volumes (all the way up to 1tb in size), and you can easily do so once you have an EBS-backed AMI.

After the EBS volume has been created, attach it to the running instance [console].  Remember what you chose as the device name the volume identified itself as (/dev/sdf for example).

In a root shell on the instance:

# mkfs.ext3 /dev/sdf

# mkdir /mnt/target && mount /dev/sdf /mnt/target

# rsync -avHx / /mnt/target

# rsync -avHx /dev /mnt/target

# sync;sync;sync;sync && umount /mnt/target

The above commands did the following:

  • formatted the entire volume /dev/sdf as an extended 3 filesystem
  • created directory /mnt/target and mounted /dev/sdf at /mnt/target
  • rsync’d the root instance-store filesystem to the ebs volume
  • synchronized the /dev directory from the instance-store filesystem
  • flush all pending write ops, and unmount the EBS volume

3) Creating the AMI – At this point, you should have a 10gb EBS volume that shows available [console].  Simply right-click on the volume and create a snapshot for the volume [console].  Once the snapshot has completed, select from the list of available kernels on ec2 with the following command:

# ./ec2-describe-images -o amazon | grep -i xenu

Store the AKI for the kernel you want to use in the environment variable AKI:

# export AKI=aki-xxxxxxxx

Up to this point, we have booted an instance-store AMI, created an EBS volume, synchronized the instance-store filesystem with the EBS volume, and created a snapshot of the EBS volume.  The only thing we need to do now is associate an AKI with the snapshot, and register the end result as an AMI in the EC2 repository.

# ./ec2-register –region eu-west-1 -s $SNAP –name $NAME –description “$DESC” –architecture $ARCH \

–root-device-name /dev/sda1

Where $SNAP is the ID of the snapshot, $NAME is the name of your AMI, $DESC is a description of the AMI, and $ARCH is either i386 (for 32-bit) or x86_64 (for 64-bit).  The command will return an AMI, which will be yours to boot from once it finishes!

To track the progress of the AMI creation, you can do the following:

# watch -n 30 ‘./ec2-describe-images ami-xxxxxxxx’

This will execute the ec2-describe-images command for your new AMI every 30 seconds.  You can stop the command once you see that the AMI is in available state.

Now that you have an EBS-backed AMI, any further customizations you make to this image can be preserved forever by simply right-clicking on the instance [console], and clicking “Create Image (EBS AMI).”

Enjoy!

No comments »

Posted in Uncategorized

Tags:

DNSSEC on the Root Zone: Are You Ready?

February 18th, 2010 by Tony No comments »

According to the ICANN, US Department of Commerce, and VeriSign, DNSSEC will be implemented on the root zones by no later than May 2010 (source: http://www.root-dnssec.org/).  For those of you who don’t know what this means, here’s a deconstruction of the sentence above:

  • DNS stands for the Domain Name System.  DNS is a hierarchical naming system for the Internet (or private network) that translates human-meaningful domain names into machine-meaningful numerical identifiers (eg, www.winnersdontlose.com -> 74.52.192.250).
  • DNSSEC stands for the Domain Name Systems Security Extensions.  It is effectively a suite of IETF-created specifications for securing certain types of data that is provided by DNS.
  • A root zone is the top-level DNS zone in the above mentioned hierarchy.  Generally speaking, “the root zone” is the largest global DNS system on the Internet.  IANA and ICANN manages this zone.  Strictly speaking, this zone is where the authorities for gTLD (.com, .net, etc) and CC TLD (.uk, .us, etc) are configured and handed out to the world.  The root zone is hosted on 13 clusters of root servers (a. to m.root-servers.net), which are the only servers in the world that are allowed to be authoritative for it.

So now, re-read what I typed.

-

According to the ICANN, US Department of Commerce, and VeriSign,
DNSSEC will be implemented on the root zones by no later than May
2010 (source: http://www.root-dnssec.org/).

-

So what does this really mean to you?  Given that the majority of us don’t need to know much more than theory about the root zone, would ICANN, US Dept of Commerce, and VeriSign really risk implementing something that could break a lot of installations today?  One would think that the majority of their decisions could be implemented in a transparent fashion, unbeknownst to the most of us. To better understand this decision, let’s look at it from two different perspectives:

  1. What benefits does DNSSEC on the root zone really provide?  Why is it necessary?
  2. What is the impact to the rest of the Internet by implementing DNSSEC on the root zone?

-

1) DNS, like a whole lot of other protocols that were created decades ago, was designed for a different Internet (lower bandwidth, higher error rate, trustworthy users).  The Internet as we know it today (including the misanthropes) has forced the creation and deprecation of many protocol specifications.  DNS suffers mostly from the trust issue.  For example, a DNS resolver currently sends a query for a resource record (RR) out to the Internet, and then will accept the first response it receives, without question.  Obviously a malicious server could provide an incorrect response, forcing the original resolver to use this incorrect address until its cache expired.  If this DNS server belongs to a major consumer ISP (like Comcat or RoadRunner), a major service interruption affecting tens of thousands of users could easily happen.

-

2) From a technology standpoint, the impact of signing the root zone has a minimal effect on end users and networks.  Since DNSSEC uses a PGP-esque keypair setup, RRs that are sent and received all day long by nameservers will be a little larger.  This will probably result in slightly larger network bills for companies managing large-ish installations, with the rest of us noticing little to no change in our operating costs.

So what IS the problem?

Part of the problem is that RFC 1035 (http://www.ietf.org/rfc/rfc1035.txt) set an upper-bound for the size (512-bytes) of a DNS response.  Because of this, many firewalls and applications used this limit in their DNS implementations.  This would not have been an issue until DNSSEC.  Since DNSSEC responses will have a signature (RRSIG) as well as the RR itself, it’s not out of the question to assume that many responses will be bigger than 512 bytes in size.

In addition to the RFC, another issue is that larger DNS packets have never been well respected on the Internet.  Most of the time, large UDP packets will get fragmented by routers along the way to the destination, and the destination either doesn’t, or can’t deal with fragmented UDP packets, and drops them.  DNS clients will usually then retry the query with a smaller buffer size, and may even eventually fail back to TCP.

-

In a nutshell, most of the problems that come about as a result of DNSSEC being implemented on the root zone are caused by larger DNS packets.  The two things you want to make sure of are:

  1. Your server/network will allow in DNS packets larger than 512-bytes.
  2. Your network/firewall will accept IP fragments, and your server knows what to do with them.

The DNS Operations Analysis and Research Center (DNS-OARC) has created a tool to help ease this process.  See https://www.dns-oarc.net/oarc/services/replysizetest for more details.

No comments »

Posted in Uncategorized

Tags:

Creating custom TCP monitors in Nagios

February 16th, 2010 by Tony No comments »

Recently I had to configure nagios to monitor a non-standard port (9999/TCP).  This is a service of ours that apparently decides to die at random times.  In order to find out more about the issue (this service is not live yet), I configured an instance of nagios to monitor this port.

In order to do this, I used the pre-existing check_tcp with a specific run-time parameter (namely, -p 9999).

First, define a command for this service check:

define command{
     command_name check_tcp_9999
     command_line $USER1$/check_tcp -h $HOSTADDRESS$ -p 9999 -4
}

Note that we manually defined the -p (port) and -h (host) parameters to use the host using this command, as well as port 9999, respectively.

Once the command has been defined, create a new service for the host in question.  If you haven’t set one up yet, below is an example to follow:

define host{
     use generic-host   ; template name, available by default
     host_name svr-001  ; unique name of the host being defined
     alias server 001   ; description of the host
     address 10.0.0.254 ; IP of the host
}

To get the service check we defined above running against this host, reference the following stanza.

define service{
     use generic-service                          ; template name, available by default
     host_name svr-001                            ; the host against which to run this check
     service_description 'tcp check of 9999/tcp'  ; self-explanatory
     check_command check_tcp_9999                 ; name of the command we defined earlier
     check_interval 5                             ; check this service every 5 minutes
     check_period 24x7                            ; time period in which to monitor
     retry_interval 3                             ; if a check fails, re-try in 3 minutes
     max_check_attempts 3                         ; the max number of times a failed service will be checked
     notification_interval 60                     ; reminder of alerts sent every 60 minutes
     notification_period 24x7                     ; time period in which alerts are sent
     notification_options w,c,u                   ; the conditions upon which to send alerts
     contact_groups contacts                      ; contact group to alert in the event of failure
}

Verify that the additions you made to the config file(s) were valid by executing:

# nagios3 -v /etc/nagios3/nagios.cfg

If the pre-flight check goes well, update your nagios installation by executing:

# /etc/init.d/nagios3 restart

That’s it!

No comments »

Posted in Uncategorized

Tags:

EC2 filesystem performance

January 26th, 2010 by Tony 1 comment »

I’ve been posting lately about a tool named bonnie++, which will run a suite of tests against your linux filesystem to determine metrics in 3 important areas:  data read/write speed, max random seeks, and max metadata operations.  Last time I posted about profiling one of Linode.com’s “Linode 360″ instances.  In this article I will profile a m1.small instance on Amazon Web Services’ (AWS) Elastic Compute Cloud (EC2) service.

EC2 is the first legitimate cloud offering to market, and in many contexts they are the most developed, most robust, cloud provider.  However, there are many companies quickly ramping up their offerings (GoGrid, Voxel, Flexiscale, etc), if only in one or two datacenters (Voxel is the leader of the group, with locations in NY, Singapore, and Amsterdam).

The m1.small instance comes with the following specifications:

  • 1.7gb RAM
  • 1 EC2 Compute Unit
  • 160gb instance storage
  • “moderate” I/O performance

While these specs are mostly useful in comparison to other EC2 instance sizes, the performance of this particular size will provide a useful benchmark for baselining EC2 performance.  Since this is the smallest instance, I’m assuming “moderate I/O performance” means that it’s as bad as it gets.

From the earlier post, we use the following command to invoke bonnie++:

# bonnie++ -u 0 -r 1700 -s 34000 -n 256 -b -d /

The above commanded failed to run, claiming that the filesystem was out of space.  Checking the filesystem, I see that I only have /dev/sda1, which is 15gb and mounted at /.  Since real-world testing involves what the customer actually gets and not what the marketing literature says, I adjusted the -S parameter to 3400, which should easily outpace/outpage the 1.7gb of memory in my instance.

Invoking bonnie++ with the new parameter yields me with the following result (click for larger image):

Server ip-10-226-125-238 was able to

  • sustain ~52MBS at 6% CPU for sequential block writes
  • sustain ~64MBS at 1% CPU for sequential block reads
  • max out at 939.8 random seeks per second
  • sequentially create 127 files per second
  • randomly create 174 files per second
  • sequentially read metadata from 158,584 files per second at 39% CPU
  • randomly read metadata from 203,851 files per second at 41% CPU
  • sequentially delete 121 files per second
  • randomly delete 158 files per second

As the results from running bonnie++ in various providers pile up, I will compile them into a spreadsheet which will (hopefully) ultimately shed some light on the performance boundaries of various VPS and cloud providers.

1 comment »

Posted in Uncategorized

Tags:

Linode filesystem performance

January 19th, 2010 by Tony No comments »

In a previous article, I wrote about using bonnie++ (http://www.coker.com.au/bonnie++/) to benchmark the performance of your hard drives and filesystem.  We went through a sample command, explained all the parameters (most notably, -s).

In this post I’d like to display and summarize the results from running bonnie++ on two systems:  a Linode (as the title suggests) and a physical server as the control.

Linode is a VPS hosting company with availability in the US and Europe.  They strive for the best service possible, and are a very well regarded provider within the hosting community.  The linode I chose for the purposes of this test is their “Linode 360″ which comes with 360MB RAM, 16GB of storage, 200GB of transit, and a price tag of $19.95 per month.  This is their entry-level setup, and their instances can scale up to 2.8GB RAM, 128GB storage, 1.6TB transit (for $159.95/month).

The command we’re running to invoke bonnie++ is:

# bonnie++ -u 0 -r 360 -s 7200 -n 256  -b -d /

Note the new values for -r and -s from my last post.

Results (click for a larger version):

On the far left of the results, in bold, you will see the machine’s hostname.  In my case, this is the Linode assigned hostname of li149-50.  The next column is labeled “Size: Chunk Size,” which is basically the size we specified as the size to use for IO performance measurements (the -s flag).

As stated in my last post, bonnie++ offers three specific metrics:

  1. Data read and write speed.
  2. Maximum seeks per second.
  3. Maximum file metadata operations per second.

Applying these three metrics to our results, the following correlations  can be established:

  • Sequential Input and Sequential Output apply to 1
  • Random Seeks apply to 2
  • Sequential Create and Random Create apply to 3

Now, let’s summarize with real numbers.

Server li149-50 was able to:

  • sustain ~85MBS at 13% CPU for sequential block writes.
  • sustain ~180MBS at 0% CPU for sequential block reads.
  • max out at 167.9 random seeks per second.
  • sequentially create 2,236 files per second.
  • randomly create 2,001 files per second.
  • sequentially read metadata from 395,067 files per second at 99% CPU.
  • randomly read metadata from 503,021 files per second at 99% CPU.
  • sequentially delete 1,014 files per second.
  • randomly delete 610 files per second.

Running the same bonnie++ command (with different -r and -s flags) on a physical server with an 80GB PATA drive yields the following (for reference, the values were “1024″ and “20480″ for “-r” and “-s”, respectively):

# bonnie++ -u 0 -r 1024 -s 20480 -n 256  -b -d /

Results (click for a larger version):

Summarizing in a similar fashion from above, we get the following:

Server tp.eliminated.org was able to

  • sustain ~41MBS at 29% CPU for sequential block writes.
  • sustain ~50MBS at 10% CPU for sequential block reads.
  • max out at 71 random seeks per second.
  • sequentially create 447 files per second.
  • randomly create 2,001 files per second.
  • sequentially read metadata from 288,111 files per second.
  • randomly read metadata from 503,021 per second.
  • sequentially delete 225 files per second.
  • randomly delete 610 files per second.

Note that the metrics regarding file metadata operations are based off of the number of files on which we test (the -n flag).  Since the value supplied to -n was the same on both servers (256), the results we get back are very similar (if not the same).

I will continue to benchmark various VPS and Cloud providers in order to ascertain IO performance from as many providers as possible.

No comments »

Posted in Uncategorized

Tags:

Benchmarking hard drives and filesystems with bonnie++

January 18th, 2010 by Tony No comments »

Bonnie++ (http://www.coker.com.au/bonnie++/) is “a benchmark suite that is aimed at performing a number of simple tests of hard drive and file system performance.”   Bonnie++ outputs a 80-column display report (to fit on braille keyboards), as well as csv values that can be converted to text or HTML (with bon_csv2txt and bon_csv2html, respectively).

Bonnie++ provides performance metrics on the following 3 things:

  1. Data read and write speed:  This should be fairly obvious.  This is how fast you can read and write to the drive(s) in question.
  2. Maximum number of seeks per second:   If the blocks that your computer needs are not sequentially stored (right next to each other), then the heads in the HD will have to do a seek (physically moves the heads to the right platter).
  3. Maximum number of file metadata operations per second:  Metadata operations include things like file creation, deletion, or gathering any other metadata about a file (permissions, size, etc).

In order to run all of the tests included, I ran the following command:

# bonnie++ -u 0 -r 1024 -s 20480 -n 256  -b -d /

Let’s break down the parameters passed to bonnie++:

  • -u 0:  The -u flag is used to indicated the UID that the test should run as.  UID 0, as we all know, belongs to the root user.
  • -r 1024:  The -r flag is used to indicate how much RAM (in megabytes) is in the machine.  As you can see I have 1gb (1024mb).
  • -s 20480:  The -s flag is used to indicate the size of the file(s) (in megabytes) to be used for IO performance testing.  More on this value later.
  • -n 256:  Specifies the number of files for the file creation test.  This is in multiples of 1024 files, so 256 = 252,144 (256*1024) files.
  • -b: The -b flag is used to disable any system-level buffering.  Basically this means an fsync() is called after every write operation.
  • -d /:  The -d flag is used to indicate the path on which to perform the tests.

Of all the runtime parameters, -s is probably the most important to define properly.  This flag (in megabytes) defines the size of the files used for IO performance benchmarking.  If the supplied size is greater than 1gb, then multiple 1gb files will be used until the value is reached.  In order to get proper results, you will want to specify a number much larger than the amount of RAM you have.  If possible, use a much higher multiple, like 20x.   To be sure to bypass any caching done by the hardware, you want at least 2-3x the amount of RAM.

No comments »

Posted in Uncategorized

Tags:

High-volume httperf

November 12th, 2009 by Tony No comments »

httperf is a great benchmarking tool that I’ve been using to do some load testing.  A colleague pointed me to it after struggling with Apache’s benchmark tool (ab).  In my opinion, httperf is by far a superior tool as it provides many more knobs and switches.  Most importantly, httperf’s output seems to me to be a little bit more usable.

One of the immediate problems I had with httperf was that whenever I had a –rate of higher than 2000, I started getting errors categorized as ‘fd-unavail’.   A little research on the Internets and the issue (as its name would indicate) is that httperf is unable to open the necessary number of file descriptors to sustain the load.  Since every open socket in Linux corresponds to an open file, the number of simultaneous open files allowed to a process or user becomes a bottleneck pretty quickly in getting httperf to perform at higher volumes.

Since my load test client was a Fedora box, the process to fix this is as follows:

1) edit /etc/security/limits.conf, adding <username> hard nofile 65535 .  <username> in the user running httperf.

  • # ulimit -n 65535

To verify the changes took place:

  • # ulimit -aH (look for “open files”)

2) edit /usr/include/bits/typesizes.h and change #define __FD_SET_SIZE 1024 to #define __FD_SET_SIZE 65535.

3) download the httperf source

  • # yum install yum-utils
  • # yumdownloader –source httperf

4) install the source RPM (you may have to make a directory)

  • # rpm -ihv httperf-0.9.0-2.fc8.src.rpm

5) compile the source RPM (and install dependency)

  • # yum install rpm-build
  • # yum install openssl-devel
  • # cd /usr/src/redhat/SPECS
  • # rpmbuild -bb httperf.specs
  • # cd ../RPMS/x86_64/

6) remove the previous install and install the new build (which can handle more than 1024 fd)

  • # yum remove httperf
  • # rpm -ihv httperf.fc8.x86_64.rpm

This will leave you with an install of httperf that can open 65535 filehandles at a time.  I did this on FC8, but I’m sure it’s easily adaptable to any RPM-based distro.

No comments »

Posted in Uncategorized

Tags:

DIY Application Acceleration Network

November 9th, 2009 by Tony 1 comment »

There is a gaping hole (which is currently being filled) in the CDN market:  application acceleration.  Application acceleration remains a bit of a holy grail for content delivery networks (CDN), mostly due to the never ending trail of patents and acquisitions in the content delivery industry.  Currently Akamai is the only company to have a hosted application acceleration network, aptly named Web Application Accelerator (WAA). 

Akamai’s approach is a generalized caching and forward mechanism layered on top of their SureRoute platform.  While this is effective en masse, there is a lot of room for targeted improvement.  The knobs and switches provided for WAA are mostly time to live (TTL) settings and other caching options.  Those of us wishing to go for a more controlled, targeted application acceleration network were historically left on our own, working through WAN replication, database sharding, and other “good enough” type solutions.  Thankfully, application acceleration and end user experience management has become a bit of a hot topic in 2008 and now the big networking companies (F5, Citrix, etc) are all developing product lines aimed at accelerating dynamic applications. A potential sleeper in this race is StrangeLoop (well, for .NET applications  anyways [for now?]), whose approach is very easily adaptable for a hosted (CDN) scenario.

In order to convey the approach behind this design I will provide some examples of products currently on the market (which is not an endorsement of any sort).

This approach requires a separation of presentation and data access.  Basically, your front-end webservers (hosting the application you’re trying to accelerate) needs to be able to access a data access tier, who in turn makes requests directly to your data store (usually a database).  This is a very common approach and allows for back-end operations to be managed independently of the front-end.  Typically, messages passed between front- and back-end servers via SOAP or something similar.  If we’re going to push these over WAN, 128-bit SSL is a minimum (alternatively, you could pipe these requests through an SSH tunnel). .

Since the end user’s experience deals largely with your front-end servers, a concept called TCP multiplexing or connection pooling decreases end user render time.  This feature is available on any layer-7 capable load balancer, like the Citrix NetScaler, or the A10 load balancers.  To most effectively utilize a WAN accelerating appliance like the Citrix WANScaler or F5’s WANJet, data access requests from presentation servers should also be multiplexed into one persistent connection back to your data access servers (so you only move large amounts of data between the database and data access, which should be physically near each other).

If possible, the interfaces used for acceleration to origin should be separate from those serving end user requests.

With this approach a company utilizing WAA can reduce their cost of goods sold by only targeting regions where business is conducted.  While there is an upfront investment of hardware, ongoing operations should be much cheaper, with a speedier end user experience.

1 comment »

Posted in Uncategorized

Tags:

Sticky Sessions and Scalability in the .NET CLR

November 9th, 2009 by Tony 1 comment »

Scaling websites that deliver static content has sadly become an old trick.  Anyone who has ever heard the letters CDN used next to each other can put together an intelligent solution for scaling static content delivery.  However, what happens when a web application that requires “sticky sessions” needs to be horizontally scaled?  Even Akamai (the only CDN with a dynamic acceleration product) necessarily forwards all of the dynamic requests back to origin.  While they do lessen the load on origin servers, the very nature of dynamic applications is such that all the processing needs to be handled by -your- servers.

Despite the fact that this entry will focus explicitly on how to do this with .NET applications, the approach to this problem is basically the same, no matter which middleware you use.

In essence, when a user signs into an application using sticky sessions, the user is assigned a cookie that identifies the session.  The server also needs to know about this, since there are server-side controls, such as session timeouts, etc.  In .NET the “session” is known as “state” and the server-side controls are governed by config files scoped by the server (machine.config) or the web app itself (web.config). 

Microsoft provides a few different ways of managing state.  At the highest level, the distinction is drawn between “InProc” (local server memory) or not.  The InProc model is a monolithic one, essentially designed for a one server + failover scenario.  While this is fine for redundancy, this will not help if your user load overwhelms that single server.  

If you do not use the InProc model, Microsoft has provided an ASP.NET state server service, which is a service that will store and retrieve state data from any ASP.NET web application.  They also provide a handful of registry keys with which to tweak the service.  I’m not a big fan of this particular mechanism, since it’s very much a “black box” approach.  I’ve found that utilizing the state server is fine most of the time, but when it doesn’t work you are more or less reduced to the traditional Windows troubleshooting method (rebooting).

An alternative for storing state data out of the memory is SQL Server.  You can take any SQL Server instance and create a special state database (the schema for which is publicly available), and then point your web applications to the SQL instance for storing and retrieving state data.  This approach is easier to troubleshoot, since you have the full array of SQL tools available, but it is much harder to setup, and is also slower than the InProc or state server models.  When I’ve used this in a production environment I found that it worked a little more reliably than the state server, but there was a very noticeable hit on performance.  (Note:  the performance impact can probably be minimized with intelligent coding but that sadly was not an option for us at the time.)

So far, I’ve written about InProc, state server, and SQL server as methods of storing and retrieving state data.  Another way of scaling your web front-end while ensuring that your end user sessions are not lost is the following:

Say I have a web application, available at http://app.company.com.  I’d like to scale this hostname such that thousands of simultaneous users will not bring my server down, but I’m also sensitive enough to end user experience such that the unreliabe or slower performance of the state server and SQL Server is not acceptable.  Consider the following flow:

1) User loads http://app.company.com.
2) The webserver for http://app.company.com does an HTTP 302 on the end user agent, and sends them to either http://app1.company.com or http://app2.company.com.
3) All further interactions with the end user happens over the hostname http://app1.company.com or http://app2.company.com.

In this scenario, we assume that machines app1 and app2 are fully contained instances of the application you are trying to scale.  These machines should be configured for InProc.  SInce we rewrote the end user’s URL in their browser, we can therefore be assured that all following interactions is with app1 or app2 directly.  The downside to this approach is the introduction of a very piece of code:  the redirector.  This code will basically have to round robin through a list of hostnames, and ideally perform some manner of a healthcheck so end users are never sent to a machine that isn’t working.  Also, while this approach does not scale forever, it does introduce extra layers of scalability which can be used (load balancing to a pair of redirector servers, etc, ad nauseum).

While some of the methodologies discussed here are specific to the .NET framework, most MW tiers have the ability to maintain state in something other than local memory.  The redirector approach is completely agnostic and should be applicable to most any scenario, but as always, none of the stuff I discuss here is an endorsement of any sort, these are more a synopsis of my experiences in having worked with this stuff for a relatively high volume environment with (what I consider to be) unreasonable demands ;)

1 comment »

Posted in Uncategorized

Tags:

IT/Sys Admin Resume

November 9th, 2009 by Tony No comments »

For this entry I thought I’d take a break from the typical SaaS performance/delivery/design entries to something a little different:  resume styles for IT / Sys Admins.  In the 10+ years I’ve been in the industry I’ve had the (dis)pleasure of interviewing something along the lines of 150-200 potential sys admin candidates, along with having to read something like 500 resumes.  I fully concede that there are others who have read/interviewed way more than I have, but since my resume gets complimented and imitated fairly often by my friends and colleagues, I thought I’d share how mine is structured.

My approach was to keep it simple, with (what I feel to be) the more relevant things up top, and scaling down to the less relevant things.  (Note: my resume is at www.eliminated.org/resume.php, feel free to provide any comments or feedback.)

—– Resume Layout —–

NAME (bigger font than anything else on the page, but not distastefully so)
contact info (in smaller font than anything else on the page)

Profile
I know a lot of people hate these “this is who I am in a nutshell” sections but I feel like this is a good place to give your audience an idea of who you are and where you see yourself going.  Anything more than a 2 or 3 sentences becomes a bit much to digest, and if you find yourself needing more than 2 or 3 sentences then maybe it’s time to re-think what it is you’d like to get across first.

Skills / Knowledge
Next to your name and how to get ahold of you, this is the next most important section for a tech resume.  By the time a company gets around to interviewing someone for a position they usually have a decent idea of which skills would be required.  Providing a brief and easy to read summary of what you know will allow someone to gather this information about you in a very little amount of time.  A little sub-categorization is nice here, depending on your focus.  For example, don’t put “Office 2000/2003″ in the same category as “Apache 2.x”- these are significantly different technologies and realistically anyone who cares about one isn’t going to care as much about the other.  On my resume, you will find the following sections: Operating Systems, Programming Languages, Software, and Services. 

Recent Experience
This section is what most people think of when they think resume.  Here, list your -relevant- job experiences, with 2-3 highlights from each job that you believe are indicative of your best/most employable qualities.  I would say that 2 or 3 of your most recent relevant jobs should be more than enough for a clear indication of what you are capable of.  A lot of times people have way too many jobs in this section, and list way too much about the specifics of what they do.  I tend to speak broadly enough that 1) someone with a non-technical background could accurately convey what I do, and 2) shows the scope of what I’ve done.  A good rule of thumb seems to be that if you can’t describe it in one sentence, then you need to re-consider what you’re trying to say.

Education, References, etc.
These sections are more there for formality than anything else.  If you have college degree, list which college, the year you graduated in, and which degree you have.  I’ve seen resumes that listed the fraternities/clubs that that someone belonged to, and frankly, it’s just not all that important in the decision making process. 

For references, a simple “References available upon request.” is more than enough.  If a company wants you bad enough they will call you and ask for the data.  This is also a good control loop to put around access to your references.  These people are (hopefully) people with whom you’ve made a positive impression; you don’t want to lose that because they are now getting spammed because of you.

If you need more space to fill, then you could put in anything else you feel might be relevant.  For example, I put down that I’m fluent in Mandarin Chinese.  This skill isn’t something I hope to have to count on for a job, but it shouldn’t hurt and really can only help.

I think one of the main problems with Sys Admin resumes that I’ve seen is that SA’s typically (or stereotypically) aren’t the kind of people who are all that great with marketing themselves.  The problem is that in order to get better jobs and to climb that proverbial ladder, you have to convince people who ARE good at marketing themselves that you too are worth it.  There are lots of ways most people could market themselves better, but at the end of the day, the first thing a prospective employer will see is your resume, and you should make that a top priority to get in order.

I welcome any feedback and suggestions that anyone reading this may have, since I don’t work in HR and don’t claim to have all the knowledge in their arena.  I do believe, however, that you should keep your resume up to date and accurate even while not searching for a new job.

No comments »

Posted in Uncategorized

Tags: