AWS CLI - Switching to and from regional EC2 reserved instances

AWS recently announced the availability of regional reserved instances, this post explains how to switch a reservation from AZ specific to regional (and back) using the AWS CLI.

Step 1, find the reservation to modify

$ aws ec2 describe-reserved-instances --filters Name=state,Values=active
{
    "ReservedInstances": [
        {
            "ReservedInstancesId": "c416aeaf-fb64-4218-970f-7426f6f32377", 
            "OfferingType": "No Upfront", 
            "AvailabilityZone": "eu-west-1c", 
            "End": "2017-10-21T08:45:55.000Z", 
            "ProductDescription": "Linux/UNIX", 
            "Scope": "Availability Zone", 
            "UsagePrice": 0.0, 
            "RecurringCharges": [
                {
                    "Amount": 0.01, 
                    "Frequency": "Hourly"
                }
            ], 
            "OfferingClass": "standard", 
            "Start": "2016-10-21T08:45:56.708Z", 
            "State": "active", 
            "FixedPrice": 0.0, 
            "CurrencyCode": "USD", 
            "Duration": 31536000, 
            "InstanceTenancy": "default", 
            "InstanceType": "t2.micro", 
            "InstanceCount": 1
        }
    ]
}

The "Scope" field in the response shows that this reservation is currently specific to an Availability Zone, eu-west-1c in this case.

Step 2, request the modification

$ aws ec2 modify-reserved-instances --reserved-instances-ids c416aeaf-fb64-4218-970f-7426f6f32377 --target-configurations Scope=Region,InstanceCount=1

{

    "ReservedInstancesModificationId": "rimod-aaada6ed-fec9-47c7-92e2-6edf7e61f2ce"

}

The Scope=Region indicates that this reservation should be converted to a regional reservation, InstanceCount is a required parameter to indicate the number of reservations the modification should be applied to.

Step 3, monitor progress

$ aws ec2 describe-reserved-instances-modifications

{

    "ReservedInstancesModifications": [

        {

            "Status": "processing", 

            "ModificationResults": [

                {

                    "ReservedInstancesId": "35f9b908-ae36-41ca-ac0b-4c67c887135b", 

                    "TargetConfiguration": {

                        "InstanceCount": 1

                    }

                }

            ], 

            "EffectiveDate": "2016-10-21T08:45:57.000Z", 

            "CreateDate": "2016-10-21T08:50:28.585Z", 

            "UpdateDate": "2016-10-21T08:50:31.098Z", 

            "ReservedInstancesModificationId": "rimod-aaada6ed-fec9-47c7-92e2-6edf7e61f2ce", 

            "ReservedInstancesIds": [

                {

                    "ReservedInstancesId": "c416aeaf-fb64-4218-970f-7426f6f32377"

                }

            ]

        }

    ]

}

The "Status" in the response will show "processing" until the modification has completed successfully, at which time it will change to "fulfilled":

$ aws ec2 describe-reserved-instances-modifications

{

    "ReservedInstancesModifications": [

        {

            "Status": "fulfilled", 

            "ModificationResults": [

                {

                    "ReservedInstancesId": "35f9b908-ae36-41ca-ac0b-4c67c887135b", 

                    "TargetConfiguration": {

                        "InstanceCount": 1

                    }

                }

            ], 

            "EffectiveDate": "2016-10-21T08:45:57.000Z", 

            "CreateDate": "2016-10-21T08:50:28.585Z", 

            "UpdateDate": "2016-10-21T09:11:33.454Z", 

            "ReservedInstancesModificationId": "rimod-aaada6ed-fec9-47c7-92e2-6edf7e61f2ce", 

            "ReservedInstancesIds": [

                {

                    "ReservedInstancesId": "c416aeaf-fb64-4218-970f-7426f6f32377"

                }

            ]

        }

    ]

}

Step 4, success!

The new reservation is now regional (Scope=Region):

$ aws ec2 describe-reserved-instances --filters Name=state,Values=active

{

    "ReservedInstances": [

        {

            "ReservedInstancesId": "35f9b908-ae36-41ca-ac0b-4c67c887135b", 

            "OfferingType": "No Upfront", 

            "FixedPrice": 0.0, 

            "End": "2017-10-21T08:45:55.000Z", 

            "ProductDescription": "Linux/UNIX", 

            "Scope": "Region", 

            "UsagePrice": 0.0, 

            "RecurringCharges": [

                {

                    "Amount": 0.01, 

                    "Frequency": "Hourly"

                }

            ], 

            "OfferingClass": "standard", 

            "Start": "2016-10-21T08:45:57.000Z", 

            "State": "active", 

            "InstanceCount": 1, 

            "CurrencyCode": "USD", 

            "Duration": 31536000, 

            "InstanceTenancy": "default", 

            "InstanceType": "t2.micro"

        }

    ]

}

Switching back

Follows the same process with the added requirement of specifying which AZ the reservation should be linked to:

$ aws ec2 modify-reserved-instances --reserved-instances-ids 35f9b908-ae36-41ca-ac0b-4c67c887135b --target-configurations Scope="Availability Zone",InstanceCount=1,AvailabilityZone=eu-west-1b

{

    "ReservedInstancesModificationId": "rimod-9e490be9-55a3-48cf-81e9-2662b13db2f8"

}

$ aws ec2 describe-reserved-instances --filters Name=state,Values=active

{

    "ReservedInstances": [

        {

            "ReservedInstancesId": "df70d097-2f33-4962-bca6-37af15ca819e", 

            "OfferingType": "No Upfront", 

            "AvailabilityZone": "eu-west-1b", 

            "End": "2017-10-21T08:45:55.000Z", 

            "ProductDescription": "Linux/UNIX", 

            "Scope": "Availability Zone", 

            "UsagePrice": 0.0, 

            "RecurringCharges": [

                {

                    "Amount": 0.01, 

                    "Frequency": "Hourly"

                }

            ], 

            "OfferingClass": "standard", 

            "Start": "2016-10-21T08:45:58.000Z", 

            "State": "active", 

            "FixedPrice": 0.0, 

            "CurrencyCode": "USD", 

            "Duration": 31536000, 

            "InstanceTenancy": "default", 

            "InstanceType": "t2.micro", 

            "InstanceCount": 1

        }

    ]

}

AWS troubleshooting - Lamba deployment package file permissions

When creating your own Lambda deployment packages be aware of the permissions on the files before zipping them. Lambda requires the files to have read access for all users, particularly "other", if this is missing you will receive a non-obvious error when trying to call the function. The fix is simple enough, perform a 'chmod a+r *' before creating your zip file. If the code is visible in the inline editor adding an empty line and saving will also fix the problem, presumably by overwriting the file with the correct permissions.

Below are some examples of errors you will see in the various languages if read permissions are missing. Hopefully this post will have saved you some time debugging.

Java CloudWatch logs:

--

Class not found: example.Hello: class java.lang.ClassNotFoundException

java.lang.ClassNotFoundException: example.Hello

at java.net.URLClassLoader$1.run(URLClassLoader.java:370)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

Caused by: java.io.FileNotFoundException: /var/task/example/Hello.class (Permission denied)

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1251)

at sun.misc.Resource.cachedInputStream(Resource.java:77)

at sun.misc.Resource.getByteBuffer(Resource.java:160)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

... 7 more

--

Java execution result (testing from console):

{

  "errorMessage": "Class not found: example.Hello",

  "errorType": "class java.lang.ClassNotFoundException"

}

Python CloudWatch logs:

--

Unable to import module 'python-hi': No module named python-hi

--

Python execution result (testing from console):

{

  "errorMessage": "Unable to import module 'python-hi'"

}

Node CloudWatch logs:

--

module initialization error: Error

    at Error (native)

    at Object.fs.openSync (fs.js:549:18)

    at Object.fs.readFileSync (fs.js:397:15)

    at Object.Module._extensions..js (module.js:415:20)

    at Module.load (module.js:343:32)

    at Function.Module._load (module.js:300:12)

    at Module.require (module.js:353:17)

    at require (internal/module.js:12:17)

--

Node execution result (testing from console):

{

  "errorMessage": "EACCES: permission denied, open '/var/task/node-hi.js'",

  "errorType": "Error",

  "stackTrace": [

    "Object.fs.openSync (fs.js:549:18)",

    "Object.fs.readFileSync (fs.js:397:15)",

    "Object.Module._extensions..js (module.js:415:20)",

    "Module.load (module.js:343:32)",

    "Function.Module._load (module.js:300:12)",

    "Module.require (module.js:353:17)",

    "require (internal/module.js:12:17)"

  ]

}

AWS Tip of the day: Tagging EC2 reserved instances

A quick post pointing out that EC2 reserved instances actually support tagging. This functionality is only available on the command line of via the API and not via the console but it still allows to you tag your reservations making it easier to keep track of why a reserved instance was purchased and what component it was intended for. Of course the reservation itself is not actually tied to a running instance in any way, it is merely a billing construct that is applied to any matching instances running in your account but if you are making architectural changes or considering different instance types for specific workloads or components the tags allow you (and your team) to see why the reservation was originally purchased. So for example if you are scaling up the instance sizes of a specific component, lets say from m4.large to m4.xlarge, you can check your reserved instance tags and modify the reservations associated with the component to ensure you continue to benefit from the purchase.

The tagging of reserved instances works the same as tagging other EC2 resources through the AWS CLI's ec2 create-tags command and specifying the reserved instances ID as the resource ID. You can find the reserved instance ID using the CLI's ec2 describe-reserved-instances command. Using an actual example, lets start off finding a reservation:

$ aws ec2 describe-reserved-instances

{

    "ReservedInstances": [

        {

            "ReservedInstancesId": "3d092b71-5243-4e5e-b409-86df342282ab", 

            "OfferingType": "No Upfront", 

            "AvailabilityZone": "eu-west-1c", 

            "End": "2017-08-12T04:48:58.000Z", 

            "ProductDescription": "Linux/UNIX", 

            "UsagePrice": 0.0, 

            "RecurringCharges": [

                {

                    "Amount": 0.01, 

                    "Frequency": "Hourly"

                }

            ], 

            "Start": "2016-08-12T04:48:59.763Z", 

            "State": "active", 

            "FixedPrice": 0.0, 

            "CurrencyCode": "USD", 

            "Duration": 31536000, 

            "InstanceTenancy": "default", 

            "InstanceType": "t2.micro", 

            "InstanceCount": 1

        }

    ]

}

Next, lets add a tag indicating that this reservation is intended for the "production" stack.:

$ aws ec2 create-tags --resources 3d092b71-5243-4e5e-b409-86df342282ab --tags Key=Stack,Value=production

Checking the result:

$ aws ec2 describe-reserved-instances

{

    "ReservedInstances": [

        {

            "ReservedInstancesId": "3d092b71-5243-4e5e-b409-86df342282ab", 

            "OfferingType": "No Upfront", 

            "AvailabilityZone": "eu-west-1c", 

            "End": "2017-08-12T04:48:58.000Z", 

            "ProductDescription": "Linux/UNIX", 

            "Tags": [

                {

                    "Value": "production", 

                    "Key": "Stack"

                }

            ], 

            "UsagePrice": 0.0, 

            "RecurringCharges": [

                {

                    "Amount": 0.01, 

                    "Frequency": "Hourly"

                }

            ], 

            "Start": "2016-08-12T04:48:59.763Z", 

            "State": "active", 

            "FixedPrice": 0.0, 

            "CurrencyCode": "USD", 

            "Duration": 31536000, 

            "InstanceTenancy": "default", 

            "InstanceType": "t2.micro", 

            "InstanceCount": 1

        }

    ]

}

Great we have a tag but what if we have hundreds of reservations, a long list of reservations is not particularly useful for quickly identifying the reservations related to a component or stack. The CLI's query and output functionality can help here:

$ aws ec2 describe-reserved-instances --query 'ReservedInstances[*].{AZ:AvailabilityZone,Type:InstanceType,Expiry:End,stack:Tags[?Key==`Stack`][?Value==`production`]}' --output=table
--------------------------------------------------------
|               DescribeReservedInstances              |
+-------------+----------------------------+-----------+
|     AZ      |          Expiry            |   Type    |
+-------------+----------------------------+-----------+
|  eu-west-1c |  2017-08-12T04:48:58.000Z  |  t2.micro |
+-------------+----------------------------+-----------+
Not quite the console view but easy enough to see that we have one reservation for the "production" Stack.

AWS Tip: Save S3 costs with abort multipart lifecycle policy

Introduction

S3 multipart uploads provide a number of benefits -- better throughput, recovery from network errors -- and a number of tools will automatically use multipart uploads for larger uploads. The AWS CLI cp, mv, and sync commands all make use of multipart uploads and make a note that "If the process is interrupted by a kill command or system failure, the in-progress multipart upload remains in Amazon S3 and must be cleaned up manually..."

The reason you would want to clean up these failed multipart uploads is because you will be charged for the storage they use while waiting for the upload to be completed (or aborted). This post provides some detail on how to find the incomplete uploads and options for removing them to save storage costs.

Finding incomplete multipart uploads

If you have relatively few buckets or only want to check your biggest buckets (CloudWatch S3 metrics are useful for finding these) the AWS CLI s3api list-multipart-uploads command is a simple check:

aws s3api list-multipart-uploads --bucket [bucket-name]

No output indicates that the bucket does not contain any incomplete uploads, see the list-multipart-uploads documentation linked above for an example of the output on a bucket that does contain an incomplete upload. A simple bash script to check all your buckets:

This script will list your buckets and the first page (out of possibly many more) incomplete multipart upload keys along with the date they were initiated. The region lookup is required to handle bucket names containing dots and buckets in eu-central-1 (SigV4).

Cleaning up

Once you have identified the buckets containing incomplete uploads it is worth investigating some of the recent failed uploads to see whether there is an underlying issue that needs to be addressed, particularly if the uploads relate to backups or important log files. The most typical cause is instances being terminated before completing uploads (look at Lifecycle Hooks to fix this if you are using Auto Scaling) but they may also be result of applications not performing cleanup on failures or not handling errors correctly.

A multipart upload can be aborted using the abort-multipart-upload s3api command in the AWS CLI using the object key and upload ID returned by list-multipart-uploads command. This can be scripted but will take time to complete for buckets containing large numbers of incomplete uploads, fortunately there is an easier way. S3 now supports a bucket lifecycle policy to automatically delete incomplete uploads after a specified period of time. Enabling the policy in the AWS console is fairly quick and easy, see Jeff's blog post for details. A rather messy boto3 example script for enabling the policy on all buckets can be found here, it should work with most bucket configurations but it comes with no guarantees and you use it at your own risk.

Conclusion (why you should do this)

If you are using S3 to store large (> 5MB) objects and you are spending more than a few dollars a month on S3 storage then there is a fairly good chance that you are paying unnecessarily for failed/incomplete multipart uploads. It should only take a few minutes to review your buckets and could potentially have significant monthly savings.

Enabling longer AWS IDs in a region using an IAM role

AWS is moving towards longer EC2 and EBS IDs and you can enable them for an IAM user or at an account level using the root credentials. You can avoid using the root credentials by using an IAM role instead. This is a quick post to explain the steps needed to use an IAM role on an instance to enable the longer IDs at an account level.

Update: You can now use the --principal-arn argument to to make this change to the root account, see the AWS support post here.

Update 2: Another alternative, using this script.

1. Create a policy allowing modify and describe of ID format (IAM console -> Policies -> Create Policy -> Create Your Own Policy). The following policy document (originally from here with the Deny changed to Allow) provides the necessary permissions,

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:ModifyIdFormat", "ec2:DescribeIdFormat"
      ],
      "Resource": "*"
    }
  ]
}

2. Create an EC2 role (IAM console -> Roles -> Create New Role -> [Pick a name] -> Amazon EC2). Select the policy name you created in step 1 and attach it to the role.

3. Launch an instance with the role. It is probably easiest to use an Amazon Linux AMI as the AWS CLI is installed by default. Instance size and region don't really matter.

4. SSH to the instance and use the CLI to enable the longer IDs in the region that you would like to test against:
$ aws --region eu-west-1 ec2 modify-id-format --resource instance --use-long-ids

If you don't specify a region the CLI will prompt you to run 'aws configure' to set a default. To check the status of the ID format you can use describe-id-format:
$ aws --region eu-west-1 ec2 describe-id-format

{
    "Statuses": [
        {
            "UseLongIds": false,
            "Resource": "reservation"
        },
        {
            "UseLongIds": true,
            "Resource": "instance"
        },
        {
            "UseLongIds": false,
            "Resource": "volume"
        },
        {
            "UseLongIds": false,
            "Resource": "snapshot"
        }
    ]
}

5. Longer IDs are now enabled for the account. Resources created by the root account (such as Auto Scaled instances) will be launched with longer IDs.

'Connection to s3.amazonaws.com timed out' using VPC S3 Endpoint

A quick post to point you in the right direction if you are getting connection time out errors using the AWS CLI or Boto with AWS S3 VPC Endpoints in a private subnet. The two most likely causes of this are:

1. Your bucket name contains dots.
To work around this you can specify the region of the bucket, for example:
aws --region eu-west-1 s3 ls s3://bucket.with.dots/

2. You are trying to access a bucket in a different region.
This is not supported by VPC Endpoints, see the restrictions here.

Memcached item memory usage

Introduction

I was recently involved with an investigation into a Memcached cluster running on AWS ElastiCache and discovered a few interesting details about Memcached. This post looks at the memory overhead of storing items in Memcached and demonstrates the impact of disabling CAS (check and set) on memory utilisation.

Background - Memcached item overhead

Before getting into the details some background might be useful. Starting with the basics, how much memory does Memcached actually need to store an item? Lets spin up a cluster to check, for simplicity I am using a single m3.large ElastiCache node (running memcached 1.4.24) which according to the AWS docs provides around 6.2GB of memory for cache items.

The easiest way to look at our new cache is by telnetting to the cluster (from within the VPC) and running commands directly. To start, what does the slab allocation look like:

> stats slabs

< STAT active_slabs 0

< STAT total_malloced 0

< END

As expected, our empty cache has 0 active slabs and no memory allocated. Lets add a value and see how this changes things:

> set k 0 500 1

> v

< STORED

Above, we have set an item with a key of 'k', no flags, a 500 second expiry time and a 1 byte value of 'v'. Looking at the slab stats we now see:

> stats slabs

< STAT 1:chunk_size 96

< STAT 1:chunks_per_page 10922

< STAT 1:total_pages 1

< STAT 1:total_chunks 10922

< STAT 1:used_chunks 1

< STAT 1:free_chunks 10921

< STAT 1:free_chunks_end 0

< STAT 1:mem_requested 67

< STAT 1:get_hits 0

< STAT 1:cmd_set 1

< STAT 1:delete_hits 0

< STAT 1:incr_hits 0

< STAT 1:decr_hits 0

< STAT 1:cas_hits 0

< STAT 1:cas_badval 0

< STAT 1:touch_hits 0

< STAT active_slabs 1

< STAT total_malloced 1048512

< END

Lots of information here but lets focus on the chunks and memory stats for the moment. We now have 1 active slab with a chunk size of 96 bytes and  one of these chunks (used_chunks) has been used to store our item, all expected except the relatively large size of the chunk used. We can see the actual memory requested was 67 bytes which means that 65 bytes of overhead are required to store the 2 byte item. Looking into the memcached code we can see that the item struct defined in memcached.h requires 48 bytes plus another 8 bytes for CAS. The remaining 11 bytes consist of:

• The null terminated key (2 bytes, 1 byte of overhead, "k\0" in our example)
• The CRLF terminated data (3 bytes, 2 bytes of overhead, "v\r\n" in our example)
• The item header which is a formatted string containing the flags, data length and terminated with CRLF (6 bytes, the minimum, in our example "_0_1\r\n" with underscores to make the spaces more visible)

The header is created in the item_make_header function in items.c, the relevant part being:

    *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);

The header formats the flags and data length as integers in a string so this overhead will increase as the data size and flag values increase. For example, setting the flag to 10 will require an extra byte of to store the extra digit in the header:

> set k 10 500 1

> v

< STORED

> stats slabs

< STAT 1:chunk_size 96

< STAT 1:used_chunks 1

< STAT 1:free_chunks 10921

< STAT 1:mem_requested 68

...

< STAT active_slabs 1

< STAT total_malloced 1048512

< END

So you can expect at least 65 bytes of overhead per item.

Disable CAS if you don't need it

CAS stands for 'Check And Set' which the Memcached protocol docs describe as: "store this data but only if no one else has updated since I last fetched it."

For a fair number of Memcached use cases CAS will not be used and as a result can be disabled with no impact. You can check your cluster using the stats command and looking at the cas_* values:

> stats

...

< STAT cas_misses 0

< STAT cas_hits 0

< STAT cas_badval 0

...

Sounds simple enough, lets test it out. After setting cas_disabled to 1 on our cluster and rebooting for the change to take effect, we set our key again:

> set k 0 500 1

> v

< STORED

> stats slabs

< STAT 1:chunk_size 96

< STAT 1:used_chunks 1

< STAT 1:free_chunks 10921

< STAT 1:mem_requested 59

...

< STAT active_slabs 1

< STAT total_malloced 1048512

< END

Not much difference except for the requested memory reducing by 8 bytes. Disabling CAS will NOT reduce the wasted memory on its own but it will allow an item to contain an extra 8 bytes of data where previously it would have needed to use a larger chunk which might be wasting significantly more memory. Lets look at an example.

Assume that you are wanting to cache an item of 32 bytes, a 16 byte key and a 16 byte value. From the background section above we know that this will take 66 bytes of overhead pushing the total item size to 98 bytes, 2 bytes more than the chunk size of the first slab (using Memcached defaults), on a CAS enabled node this shows:

> set 0123456789ABCDEF 0 500 16

> 0123456789ABCDEF

< STORED

> stats slabs

< STAT 2:chunk_size 120

< STAT 2:used_chunks 1

< STAT 2:free_chunks 8737

< STAT 2:mem_requested 98

...

< STAT active_slabs 1

< STAT total_malloced 1048560

< END

Whereas with CAS disabled the overhead is reduced to 58 bytes, allowing the item to fit into the original slab:

> set 0123456789ABCDEF 0 500 16

> 0123456789ABCDEF

< STORED

> stats slabs

> STAT 1:chunk_size 96

> STAT 1:used_chunks 1

> STAT 1:free_chunks 10921

> STAT 1:mem_requested 90

...

> STAT active_slabs 1

> STAT total_malloced 1048512

> END

This means that with CAS enabled we are wasting 18% of the memory in each chunk of this slab (22 out of 120 bytes lost per item) compared to just 6% (6 out of 96 bytes) lost with CAS disabled. If the entire cache consisted of items this size, it would result in more than 1 GB of wasted memory compared to 372 MB wasted with CAS disabled. To put this differently, disabling CAS would allow you to store approximately 7 million additional 90 byte items resulting in better cache hit rates and lower eviction rates. This is of course an artificial example so lets look at a more realistic example where the cache item sizes are not quite so uniform.

The following stats are based on a production CAS enabled node's stats adjusted by a factor of ten to fit on an m3.large (the production node had around 59 million items in slab 11 this was scaled to 5.9 million for testing). The test code used to generate this data is relatively simple and distributes the items in a slab fairly evenly across the item sizes of a slab, each item's size is slightly weighted to try and match the average item size in production.

Slab	Chunk Size	Chunks Used	Memory Allocated	Memory Requested	Average Item Size	Memory Wasted
4	192	139,810	26,843,520	25,706,546	184	1,136,974
5	240	55,924	13,421,760	11,399,776	204	2,021,984
10	752	232,025	174,482,800	162,673,515	701	11,809,285
11	944	5,985,735	5,650,533,840	4,643,546,639	776	1,006,987,201
12	1,184	10,951	12,965,984	11,429,965	1,044	1,536,019
13	1,480	7,177	10,621,960	9,421,613	1,313	1,200,347
14	1,856	6,071	11,267,776	10,148,664	1,672	1,119,112
15	2,320	5,325	12,354,000	11,094,291	2,083	1,259,709
16	2,904	4,621	13,419,384	11,947,574	2,585	1,471,810
17	3,632	3,695	13,420,240	11,906,634	3,222	1,513,606
18	4,544	5,201	23,633,344	21,025,609	4,043	2,607,735
19	5,680	4,523	25,690,640	23,038,012	5,094	2,652,628
20	7,104	3,778	26,838,912	23,778,017	6,294	3,060,895
21	8,880	3,022	26,835,360	23,630,979	7,820	3,204,381
22	11,104	2,417	26,838,368	23,754,944	9,828	3,083,424
23	13,880	1,933	26,830,040	23,092,970	11,947	3,737,070
24	17,352	1,107	19,208,664	15,554,075	14,051	3,654,589
		6,473,315	6,115,206,592	5,063,149,823		1,052,056,769

The table shows that we are caching around 6.4 million items with the majority of them falling in slabs 4, 10, and 11 and that around 1 GB of memory is being wasted. Average item size is computed from (memory requested / chunks used, rounded to the nearest byte) and is an indicator of the actual item sizes (including overhead) within each slab. An interesting point relevant to the CAS discussion is that the average item size for slab 11 is 776 bytes which is only marginally bigger than the chunk size for slab 10.

Running the test code against a node with CAS disabled yields the following results:

Slab	Chunk Size	Chunks Used	Memory Allocated	Memory Requested	Average Item Size	Memory Wasted
4	192	165,990	31,870,080	29,471,729	178	2,398,351
5	240	29,744	7,138,560	6,068,721	204	1,069,839
10	752	3,365,926	2,531,176,352	2,496,542,132	742	34,634,220
11	944	2,853,079	2,693,306,576	2,261,103,795	793	432,202,781
12	1,184	10,331	12,231,904	10,910,829	1,056	1,321,075
13	1,480	6,603	9,772,440	8,703,301	1,318	1,069,139
14	1,856	6,116	11,351,296	10,202,507	1,668	1,148,789
15	2,320	5,469	12,688,080	11,429,195	2,090	1,258,885
16	2,904	4,651	13,506,504	12,137,620	2,610	1,368,884
17	3,632	3,629	13,180,528	11,834,384	3,261	1,346,144
18	4,544	5,089	23,124,416	20,661,849	4,060	2,462,567
19	5,680	4,557	25,883,760	23,299,194	5,113	2,584,566
20	7,104	3,786	26,895,744	23,983,965	6,335	2,911,779
21	8,880	3,056	27,137,280	24,146,469	7,901	2,990,811
22	11,104	2,376	26,383,104	23,654,235	9,955	2,728,869
23	13,880	2,297	31,882,360	28,479,830	12,399	3,402,530
24	17,352	616	10,688,832	8,733,548	14,178	1,955,284
		6,473,315	5,508,217,816	5,011,363,303		496,854,513

As expected, the extra 8 bytes per slab has allowed a large number of items to be moved to slabs having smaller chunk sizes with more than half of the items in slab 11 now fitting into slab 10 and saving almost 600 MB of memory. The average item size being closer to the slab chunk size is another indicator that we are making more efficient use of the memory in each slab.

For reference, the size of the items each slab can contain are below. The chunk sizes are displayed when running 'memcached -vv', reformatting this into a table and calculating the maximum item size gives:

Slab	Chunk size	Max data	Min data
1	96	30	1
2	120	54	31
3	152	86	55
4	192	125	87
5	240	173	126
6	304	237	174
7	384	317	238
8	480	413	318
9	600	533	414
10	752	685	534
11	944	877	686
12	1184	1116	878
13	1480	1412	1117
14	1856	1788	1413
15	2320	2252	1789
16	2904	2836	2253
17	3632	3564	2837
18	4544	4476	3565
19	5680	5612	4477
20	7104	7036	5613
21	8880	8812	7037
22	11104	11035	8813
23	13880	13811	11036
24	17352	17283	13812
25	21696	21627	17284
26	27120	27051	21628
27	33904	33835	27052
28	42384	42315	33836
29	52984	52915	42316
30	66232	66163	52916
31	82792	82723	66164
32	103496	103426	82724
33	129376	129306	103427
34	161720	161650	129307
35	202152	202082	161651
36	252696	252626	202083
37	315872	315802	252627
38	394840	394770	315803
39	493552	493482	394771
40	616944	616874	493483
41	771184	771114	616875
42	1048576	1048505	771115

This table is using the default Memcached config settings (48 min item, 1.25 growth factor, cas enabled). The "Max data" column is the maximum number of bytes that can be used to store both the key and the value for an item in this slab. The "Min data" column is the minimum data size required for an item to be put in a slab and is simply the maximum data size from the previous slab + 1. With CAS disabled the max and min data values would be increased by 8 bytes for all items with the exception of the minimum data for slab 1 remaining at 1 byte. The following Python function can be used for calculating the maximum item size (key + value) per slab:

Conclusion

In this post we have had a look at the details of Memcached item overhead and identified disabling CAS as a simple mechanism of improving memory utilisation which will reduce evictions and increase hit rate. In the next post we will look at how adjusting the growth factor and minimum item size can further improve memory utilisation.