tag:blogger.com,1999:blog-80823024453060013552024-02-19T10:19:51.130+00:00DeplicationSimpler is better.Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comBlogger54125tag:blogger.com,1999:blog-8082302445306001355.post-15068689183183923812021-08-08T10:44:00.000+01:002021-08-08T10:44:48.151+01:00Reducing VPC interface endpoint costs in dev/test environments<p>Two quick tips for potentially reducing VPC interface endpoint costs.</p><p>Firstly an Athena query for finding unused VPC interface endpoints in cost and usage report data. This requires you to have <a href="https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html">cost and usage reporting</a> enabled (you should already) and configured for querying in Athena, see <a href="https://docs.aws.amazon.com/cur/latest/userguide/cur-query-athena.html">this guide</a>. The query searches for all VPC interface endpoints that have not accrued any data transfer charges during the query period. VPC endpoints incur both hourly and data transfer <a href="https://aws.amazon.com/privatelink/pricing/">charges</a>, if you are not transferring any data it is very likely the endpoint is not being used and can be deleted. This will save around $7 a month ($0.01 x 24hours x 30 days) in each availability zone configured for the unused endpoint (by default all AZs). Annual cost saving potential of around $260 for each (3 AZ) endpoint removed in each account.</p><div style="text-align: left;">Query:<br /><div>select line_item_usage_account_id, line_item_resource_id</div>from cur<br />where line_item_usage_type like '%VpcEndpoint-Hour%'<br />and year='2021'<br />and month in ('7','07')<br />and line_item_resource_id not in <br /> (select distinct(line_item_resource_id) <br /> from cur<br /> where line_item_usage_type like '%VpcEndpoint-Byte%'<br /> and year='2021'<br /> and month in ('7','07'))</div><div style="text-align: left;">group by line_item_usage_account_id, line_item_resource_id</div><p>The second approach is only recommended for dev/test environments where availability zone fault tolerance is not a requirement. When you create a VPC interface endpoint an ENI is created in each subnet (availability zone) unless you explicitly deselect the subnets. This is a best practice for interface endpoints as it provides continued availability in the case of an AZ failure - as long as you are using the (default) regional interface endpoint DNS name with <a href="https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-interface.html#vpce-private-dns">private DNS</a>. When using private DNS the regional endpoint DNS name, ec2.eu-west-1.amazonaws.com for example, is resolved to the private IP of the VPC interface endpoint ENI rather than the public IP of the service. In the case where one AZ is not reachable the request will be routed to an interface endpoint ENI in a different AZ, allowing your client applications to continue functioning. The down side of this is that you will incur intra-region data transfer charges (of $0.01/GB) for sending requests to the ENI in a different AZ. The potential cost saving opportunity is to remove this availability mechanism to reduce the hourly interface endpoint charges. </p><p>As long as the data transferred to and from the interface endpoint is around 1GB/hour it is cheaper to configure the VPC interface endpoint to only be available in one (or two) availability zones. This makes sense for any API specific endpoints (EC2 for example) but may not make sense for high throughput data intensive endpoints (such as S3/Kinesis/Cloudwatch Logs). The cost and usage report data can once again be used to help identify interface endpoints where the data transfer is low enough to justify exposing the endpoint in only one availability zone. Note that the endpoint data charges are aggregated across all ENIs so the query will not correctly identify cases where traffic is not more or less equally balanced across AZs. The query assumes that the cost and usage report is configured for hourly granularity, the line_item_usage_amount would need to be modified for daily/monthly reports and would be less accurate. Annual cost saving potential would depend on data transfer and number of AZs provisioned but should be around $175 for API endpoints reduced from 3 AZs to 1 AZ.</p><div style="text-align: left;">Query:<br />select line_item_usage_account_id, line_item_resource_id<br />from cur<br />where line_item_usage_type like '%VpcEndpoint-Byte%'<br /> and year='2021'<br /> and month in ('7','07')<br /> and line_item_usage_amount < 1.0<br /> and line_item_resource_id not in <br /> (select distinct(line_item_resource_id) <br /> from cur<br /> where line_item_usage_type like '%VpcEndpoint-Byte%'<br /> and year='2021'<br /> and month in ('7','07')<br /> and line_item_usage_amount > 1.0)<br />group by line_item_usage_account_id, line_item_resource_id</div>Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-555118721161608992021-07-01T09:52:00.000+01:002021-07-01T09:52:54.661+01:00Checking regional service availability with the AWS CLI<div style="text-align: left;">Using <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-public-parameters-global-infrastructure.html">System Manager public parameters</a> a quick CLI one liner for checking if an AWS service is available in a region:</div><div style="text-align: left;"><i>aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/services/[SERVICE]/regions/[REGION] --output text</i></div><div style="text-align: left;"><br /></div><div style="text-align: left;">As an example, checking if Athena is available in ap-northeast-3:</div><div style="text-align: left;"><i>aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/services/athena/regions/ap-northeast-3 --output text</i></div><div style="text-align: left;"><i><br /></i></div><div style="text-align: left;">An empty result indicates the service is not available. The list of service names can be retrieved as below (from the documentation linked above):</div><div style="text-align: left;"><div><i>aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/services --query 'Parameters[].Name | sort(@)'</i></div><div><br /></div><div>Similarly the list of regions can be found using:</div><div><div><i>aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/regions --query 'Parameters[].Name'</i></div></div></div>Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-51866936638186067952019-09-07T12:56:00.002+01:002019-09-07T12:58:01.849+01:00AWS in the weeds - S3 CloudWatch metrics and lifecycle actions that are in progressI was recently working with a customer on an issue that highlighted a poorly explained side effect of the reversibility of lifecycle actions. The purpose of this post is to explain this behaviour with the hope that it will save S3 customers unexpected costs. TLDR: S3 CloudWatch metrics don't accurately display metrics about lifecycle actions that are still in progress.<br />
<br />
The customer use case was fairly simple, they had a bucket with a large amount of data that needed to be deleted. They added a <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#intro-lifecycle-rules-actions">lifecycle expiration action</a> to the bucket and the following day noticed that the CloudWatch bucket size metric showed the bucket size to have reduced by the expected amount. An example of how this looks in CloudWatch is below, this graph is a reproduction in my own account, the customer had a significantly larger amount of data to delete (petabytes rather than terabytes):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk1PVB1gYCpJbcVqjEVr83jdZ_ILHPb5BeSvgjmoG6b6bs2EeqgLdwrkoqRYIjiLe-lVs9ffabtTWfpq26GKB_U0GTsprZBPI3Pd4XR2pykEr8KZl5_b9l-4dPXDr0H5zaUoYMxdjojSA/s1600/mistaken.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="495" data-original-width="493" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk1PVB1gYCpJbcVqjEVr83jdZ_ILHPb5BeSvgjmoG6b6bs2EeqgLdwrkoqRYIjiLe-lVs9ffabtTWfpq26GKB_U0GTsprZBPI3Pd4XR2pykEr8KZl5_b9l-4dPXDr0H5zaUoYMxdjojSA/s320/mistaken.png" width="318" /></a></div>
<br />
The customer assumed (logically) that the drop in the graph meant that the lifecycle action had completed and they removed the expiration lifecycle action. A couple of weeks later the customer's finance team noticed that their S3 storage costs had not reduced despite the engineering team telling them to expect a significant decrease. Looking at the relevant bucket the CloudWatch graph showed that the bucket size reduction had been reverted and most of the supposedly deleted data was still present. Example graph:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbicCXItjfs768KBURlA5GYrfZIpJPZBBrlXazNezP6e9et0aj0P-fh1UhgYSFsIILna1z_JGC1KTdWy4vimQZn9YK6yVoCmIFzETu41Ae1ABrWhJ-ooVckgN7FHFlmRsX2d8SGX_6Y7I/s1600/bounce.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="487" data-original-width="462" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbicCXItjfs768KBURlA5GYrfZIpJPZBBrlXazNezP6e9et0aj0P-fh1UhgYSFsIILna1z_JGC1KTdWy4vimQZn9YK6yVoCmIFzETu41Ae1ABrWhJ-ooVckgN7FHFlmRsX2d8SGX_6Y7I/s320/bounce.png" width="303" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
After investigating with the customer it became clear that this was an unintended consequence of the way S3 lifecycle actions are <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-set-lifecycle-configuration-intro.html">implemented</a>, specifically that:<br />
<br />
"When you disable or delete a lifecycle rule, after a small delay Amazon S3 stops scheduling new objects for deletion or transition. Any objects that were already scheduled will be unscheduled and they won't be deleted or transitioned."<br />
<br />
The documentation unfortunately neglects to mention that the CloudWatch bucket metrics will update immediately even when the lifecycle action has not actually completed processing. There is currently no way to view progress on the lifecycle action and in this case it took the better part of 2 weeks for the action to complete. This was a fairly expensive lesson for the customer as S3 was 'operating as designed' and the customer was charged for the data storage costs from the time they removed the lifecycle action until they added it back when they noticed the issue.<br />
<br />
Takeaways to help avoid this problem:<br />
1. Lifecycle actions can take a long time to complete, potentially weeks for large (petabyte) volumes of data.<br />
2. Don't used CloudWatch bucket metrics as an indicator of lifecycle action progress or completion.<br />
3. After removing or disabling a lifecycle action make sure you check the CloudWatch metrics the next day or two to confirm there is no unexpected reversion of the action.<br />
4. You will be charged for data costs if the removal of the lifecycle action results in the action being reverted for some objects.Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-73805659578412212242019-02-16T15:21:00.000+00:002019-02-16T15:21:40.390+00:00AWS tip: Wildcard characters in S3 lifecycle policy prefixesA quick word of warning regarding S3's treatment of asterisks (*) in object <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html">lifecycle policies</a>. In S3 asterisks are valid <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-keys">'special' characters</a> and can be used in object key names, this can lead to a lifecycle action not being applied as expected when the prefix contains an asterisk.<br />
<br />
Historically an asterisk is treated as a wildcard to pattern match 'any', so you would be able to conveniently match all files for a certain pattern: 'rm *' as an example, would delete all files. This is NOT how an asterisk behaves in S3 lifecycle prefixes. If you specify a prefix of '*' or '/*' it will only be applied to objects that start with an asterisk and not all objects. The '*' prefix rule would be applied to these objects:<br />
example-bucket/*/key<br />
example-bucket/*yek<br />
<br />
But would not be applied to:<br />
example-bucket/object<br />
example-bucket/directory/key<br />
example-bucket/tcejbo*<br />
<br />
It is not an error to specify an asterisk and it will merely result in the policy not being applied so you may not even know this as an issue. Fortunately it is fairly easy to check for this configuration with the CLI, the following bash one liner will iterate through all buckets owned by the caller and check if the bucket has any policies with an asterisk in their name. It will print out the bucket name and policy if affected or a '.' (to show progress) if not.<br />
<br />
<script src="https://gist.github.com/watchamcb/bc99754981bc68b5c398829ca51b72be.js"></script>
<div>
You can also check the S3 console, it should display 'Whole bucket' under the 'Applied To' column for any lifecycle rules you intended to have applied to the entire bucket.</div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-40925948160222013762018-10-14T12:14:00.002+01:002019-02-14T21:22:53.384+00:00Finding S3 API requests from previous versions of the AWS CLI and SDKs<div class="separator" style="clear: both; text-align: center;">
</div>
<h3>
Introduction</h3>
Earlier this year the S3 team <span class="MsoHyperlink"><a href="https://forums.aws.amazon.com/ann.jspa?annID=5816">announced</a></span>
that S3 will stop accepting API requests <span class="MsoHyperlink"><a href="https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html">signed</a></span>
using AWS Signature Version 2 after June 24th, 2019. Customers will need to
update their SDKs, CLIs, and custom implementations to make use of <span class="MsoHyperlink"><a href="https://docs.aws.amazon.com/general/latest/gr/sigv4_changes.html">AWS
Signature Version 4</a></span> to avoid impact after this date. It might be
difficult to find older applications or instances using outdated versions of
the AWS CLI or SDKs that need to be updated, the purpose of this post is to
explain how AWS CloudTrail <span class="MsoHyperlink"><a href="https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-and-data-events-with-cloudtrail.html#logging-data-events">data
events</a></span> and Amazon Athena can be used to help identify applications
that may need to be updated. We will cover the setup of the CloudTrail data
events, the Athena table creation, and some Athena queries to filter and refine
the results to help with this process.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<h3>
Update (January/February 2019)</h3>
<div>
S3 <a href="https://forums.aws.amazon.com/ann.jspa?annID=6551">recently added</a> a SignatureVersion item to the AdditionalEventData field in the S3 data events, this significantly simplifies the process of finding clients using SigV2. The SQL queries below have been updated to exclude events with a SigV4 signature (additionaleventdata NOT LIKE '%SigV4%'). You can equally search for only '%SigV2%' and skip the CLI version string munging entirely.</div>
<div>
<br /></div>
<div class="MsoNormal">
<o:p></o:p></div>
<h3>
Setting up CloudTrail data events in the AWS console</h3>
<h1>
<o:p></o:p></h1>
<div class="MsoNormal">
The first step is to create a trail to capture S3 data
events. This should be done in the region you plan on running your Athena
queries in order to avoid unnecessary data transfer charges. In the CloudTrail console
for the region, create a new trail specifying the trail name. The ‘Apply trail
to all regions’ option should be left as ‘Yes’ unless you plan on running
separate analyses for each region. Given that we are creating a data events
trail, select ‘None’ under the Management Events section and check the “Select
all S3 buckets in your account” checkbox. Finally select the S3 location where
the CloudTrail data will be written, we will create new bucket for simplicity:<o:p></o:p><br />
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicqyJR3bVMAK9WZJje_DhhmUibchHe63cewImTcKtavLOQW9aEkJN460Xch5-hsycJJ6aHf00MwxKFwa0muLfbn5F5ElogevzGe_D_V6T0hxaSIDyhlxnoveNfaybNKLi3bD9MbNiToOU/s1600/create_trail.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="873" data-original-width="1128" height="494" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicqyJR3bVMAK9WZJje_DhhmUibchHe63cewImTcKtavLOQW9aEkJN460Xch5-hsycJJ6aHf00MwxKFwa0muLfbn5F5ElogevzGe_D_V6T0hxaSIDyhlxnoveNfaybNKLi3bD9MbNiToOU/s640/create_trail.png" width="640" /></a></div>
<br />
<div class="MsoNormal">
<br /></div>
<h3>
Setting up CloudTrail data events using the AWS CLI</h3>
<h1>
<o:p></o:p></h1>
<div class="MsoNormal">
If you prefer to create the trail using the AWS CLI then you
can use the <span class="MsoHyperlink"><a href="https://docs.aws.amazon.com/cli/latest/reference/cloudtrail/create-subscription.html">create-subscription</a></span>
command to create the S3 bucket and trail with the correct permissions,
updating it to be a global trail and then adding the S3 data event
configuration:<o:p></o:p></div>
<script src="https://gist.github.com/watchamcb/6afc2efe1e6b287d4007c17e696cfc1c.js"></script>
<br />
<div class="MsoNormal">
<br />
<br /></div>
<h3>
A word on cost</h3>
<h1>
<o:p></o:p></h1>
<div class="MsoNormal">
Once the trail has been created, CloudTrail will start
recording S3 data events and delivering them to the configured S3 bucket. Data
events are currently priced at $0.10 per 100,000 events with the storage costs
being the standard S3 data storage charges for the (compressed) events, see the <span class="MsoHyperlink"><a href="https://aws.amazon.com/cloudtrail/pricing/">CloudTrail pricing</a></span> page for additional details. It is recommend that you disable the data event trail once you are satisfied that you have gathered sufficient
request data, it can be re-enabled if further analysis is required at a later stage.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<h3>
Creating the Athena table</h3>
<h1>
<o:p></o:p></h1>
<div class="MsoNormal">
The CloudTrail team simplified the process for using Athena
to analyse CloudTrail logs by <span class="MsoHyperlink"><a href="https://aws.amazon.com/about-aws/whats-new/2018/03/aws-cloudtrail-log-search-using-amazon-athena/">adding
a feature</a></span> to allow customers to create an Athena table directly from
the CloudTrail console event history page by simply clicking on the ‘Run
advanced queries in Amazon Athena’ link and selecting the corresponding S3
CloudTrail bucket:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9yd9A9WTbwlw_y5Cb30_efrqjg3x3KZDZDkq32rQkIGn7hUVc_8bjvjJTpYnZny6xdFlHTM3B4RYVfBuroR6hj1fKuKiHUut6ZLc-sNPnRB5LbMV3jd17MYRFjGMP6D16_5km1-hGHS8/s1600/create_table.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="277" data-original-width="1329" height="132" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9yd9A9WTbwlw_y5Cb30_efrqjg3x3KZDZDkq32rQkIGn7hUVc_8bjvjJTpYnZny6xdFlHTM3B4RYVfBuroR6hj1fKuKiHUut6ZLc-sNPnRB5LbMV3jd17MYRFjGMP6D16_5km1-hGHS8/s640/create_table.png" width="640" /></a></div>
<br />
<br />
<div class="MsoNormal">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2hrhGinV_7AQJNtF3h7AY56L_XemUFifhgjoxGNgGaHeu2KyMMrwlg9t6erxHQa0qRSiCphbqmnrGHnjDPmT5cGgTnlLcDh6UqzYFfUu03Wa53idDKh6hLDUfSttL6XBYx-UWOlXE1QQ/s1600/enable_athena.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="790" data-original-width="806" height="626" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2hrhGinV_7AQJNtF3h7AY56L_XemUFifhgjoxGNgGaHeu2KyMMrwlg9t6erxHQa0qRSiCphbqmnrGHnjDPmT5cGgTnlLcDh6UqzYFfUu03Wa53idDKh6hLDUfSttL6XBYx-UWOlXE1QQ/s640/enable_athena.png" width="640" /></a></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
An explanation of how to create the Athena table manually
can be found in the <span class="MsoHyperlink"><a href="https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html#create-cloudtrail-table-ct">Athena
CloudTrail documentation</a></span>.<o:p></o:p><br />
<br /></div>
<h3>
Analysing the data events with Athena</h3>
<h1>
<o:p></o:p></h1>
<div class="MsoNormal">
We now have all the components needed to begin searching for
clients that may need to be updated. Starting with a basic query that filters
out most of the AWS requests (for example the AWS Console, CloudTrail, Athena, Storage
Gateway, CloudFront):<o:p></o:p></div>
<br />
<script src="https://gist.github.com/watchamcb/f544054890f86dca6a43748e83405ec4.js"></script>
<br />
<div class="MsoNormal">
<o:p><br /></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8SjfozLYB5VoM-t9j6sYnRBTy-EOiwUzv-V5O60NyY38osOUQj18ujaVlIsC97yGtaJcsYaV5PjWYCXuxXEzftunjWEdyhxYWkXhAwnXuQ47wUK0pJ3lOOUdrGip7DwV2Yd6rQdOYH68/s1600/query2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="502" data-original-width="867" height="370" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8SjfozLYB5VoM-t9j6sYnRBTy-EOiwUzv-V5O60NyY38osOUQj18ujaVlIsC97yGtaJcsYaV5PjWYCXuxXEzftunjWEdyhxYWkXhAwnXuQ47wUK0pJ3lOOUdrGip7DwV2Yd6rQdOYH68/s640/query2.png" width="640" /></a></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
These results should mostly be client API/CLI requests but the
large number of requests can still be refined by only including regions that
actually support AWS Signature Version 2. From the <span class="MsoHyperlink"><a href="https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region">region
and endpoint documentation for S3</a></span> we can see that we only need to
check eight of the regions. We can safely exclude the AWS Signature Version 4
(SigV4) regions as clients would not work correctly against these regions if
they did not already have SigV4 support. Let’s also look at distinct user
agents and extract the version from the user agent string:<o:p></o:p></div>
<br />
<script src="https://gist.github.com/watchamcb/448e7a0f9419dca53392bfd20198527e.js"></script>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9N32SwpunF63o4vlfC4FdzpulJXiyLczFFPDIksluBfMF8D1a8OMdjVlqvphZQRrwbpnvOknfJC99a6TNR9Le78GvlxiqanMnFDychXHIeEuDDbezH9E6FusPwd3RMOWqX-OX58z4TxI/s1600/query3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="497" data-original-width="919" height="346" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9N32SwpunF63o4vlfC4FdzpulJXiyLczFFPDIksluBfMF8D1a8OMdjVlqvphZQRrwbpnvOknfJC99a6TNR9Le78GvlxiqanMnFDychXHIeEuDDbezH9E6FusPwd3RMOWqX-OX58z4TxI/s640/query3.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
We are unfortunately not able to filter on the calculated
‘version’ column and as it is a string it is also difficult to perform direct
numerical version comparison. We can use some arithmetic to create a version
number that can be compared. Using the AWS CLI requests as an example for the moment and
adding back the source IP address and user identity<o:p></o:p></div>
<br />
<script src="https://gist.github.com/watchamcb/4759d6bc9f1dc8b1304590c0be793972.js"></script>
<br />
<div class="MsoNormal">
The version comparison number (10110108) translates to the
version string 1.11.108 which is the first version of AWS CLI supporting SigV4
by default. This results in a list of clients accessing S3 objects in this
account using a version of the AWS CLI that needs to be updated:<o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdmsDfVaHUK-6pJ4-HjEopfiaW3uZagAN4gu3_nZUSCzF3J5C3rKPsKfpWjkZNnQdXMMQZVN9sQUAyhaAykkhfGYSmlI95PC-nx54Nhk7BMVle2abpRXnTSXpSvltt05MbAIYnhkJZbGQ/s1600/query4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="429" data-original-width="920" height="298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdmsDfVaHUK-6pJ4-HjEopfiaW3uZagAN4gu3_nZUSCzF3J5C3rKPsKfpWjkZNnQdXMMQZVN9sQUAyhaAykkhfGYSmlI95PC-nx54Nhk7BMVle2abpRXnTSXpSvltt05MbAIYnhkJZbGQ/s640/query4.png" width="640" /></a></div>
<br />
<br />
<div class="MsoNormal">
The same query can be applied to all the AWS CLI and SDK
user agent strings by substituting the corresponding agent string and version
number for SDK versions using SigV4 by default:<o:p></o:p><br />
<br /></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0cm 5.4pt 0cm 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="height: 15.0pt; mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td nowrap="" style="border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
<b>AWS Client<o:p></o:p></b></div>
</td>
<td nowrap="" style="border-left: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
<b>SigV4 default version<o:p></o:p></b></div>
</td>
<td nowrap="" style="border-left: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
<b>User Agent String<o:p></o:p></b></div>
</td>
<td nowrap="" style="border-left: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
<b>Version comparator<o:p></o:p></b></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 1;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Java<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
1.11.x<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
aws-sdk-java<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
10110000<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 2;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
.NET<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
3.1.10.0<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
aws-sdk-dotnet<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
30010010<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 3;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Node.js<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
2.68.0<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
aws-sdk-nodejs<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
20680000<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 4;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
PHP<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
3<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
aws-sdk-php<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
30000000<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 5;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Python Botocore<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
1.5.71<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Botocore<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
10050071<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 6;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Python Boto3<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
1.4.6<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Boto3<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
10040006<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 7;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Ruby<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
2.2.0<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
aws-sdk-ruby<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
20020000<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 8;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
AWS CLI<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
1.11.108<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
aws-cli<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
10110108<o:p></o:p></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 9; mso-yfti-lastrow: yes;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 99.0pt;" valign="top" width="132"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
Powershell<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 89.0pt;" valign="top" width="119"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
3.1.10.0<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 86.0pt;" valign="top" width="115"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
AWSPowerShell<o:p></o:p></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.95pt;" valign="top" width="88"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm;">
30010010<o:p></o:p></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Note:<o:p></o:p></div>
<div class="MsoNormal">
.NET35,.NET45, and
CoreCLR only, PCL, Xamarin, UWP platforms do not support SigV4 at all<o:p></o:p></div>
<div class="MsoNormal">
All versions of Go and C++ SDKs support SigV4 by default<o:p></o:p></div>
<div class="MsoNormal">
<br />
Additional Note:<br />
There is no need to look at the client version number for new events which will automatically include the SignatureVersion.<br />
<br /></div>
<h3>
Tracing the source of the requests</h3>
<h1>
<o:p></o:p></h1>
<div class="MsoNormal">
The source IP address will reflect the private IP of the EC2
instance accessing S3 through a VPC endpoint or the public IP if accessing S3
directly. You can search for either of these IPs in EC2 AWS Console for the
corresponding region. For non-EC2 or NAT access you should be able to use the
ARN to track down the source of the requests.<o:p></o:p></div>
<br />Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-69225621503438941082018-08-25T17:23:00.002+01:002021-07-01T10:34:00.523+01:00AWS S3 event aggregation with Lambda and DynamoDB<h4>
Introduction</h4>
S3 has had <a href="https://aws.amazon.com/blogs/aws/s3-event-notification/">event notifications</a> since 2014 and for <a href="https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html">individual object notifications</a> these events work well with Lambda, allowing you to perform an action on every object event in a bucket. It is harder to use this approach when you want to perform an action a limited number of times or at an aggregated bucket level. An example use case would be refreshing a dependency (like <a href="https://docs.aws.amazon.com/storagegateway/latest/userguide/managing-gateway-file.html#refresh-cache">Storage Gateway RefreshCache</a>) when you are expecting a large number of objects events in a bucket. Performing a relatively expensive action for every event is not practical or efficient in this case. This post provides a solution for aggregating these events using Lambda, DynamoDB, and SQS.<div><br /><h4 style="text-align: left;">Update (June/July 2020)</h4><div>Cache refresh can now be <a href="https://aws.amazon.com/blogs/storage/automating-cache-refresh-process-for-file-gateway-on-aws-storage-gateway/">automated</a>. The event aggregation approach below may still be useful depending on your requirements.</div><h4><br /></h4><h4>
The problem</h4>
<div>
We want to call RefreshCache on our Storage Gateway (SGW) whenever the contents of the S3 bucket it exposes are updated by an external process. If the external process is updating a large number of (small) S3 objects then a large number of S3 events will be triggered. We don't want to overload our SGW with refresh requests so we need a way to aggregate these events to only send occasional refresh requests.</div>
<div>
<br /></div>
<h4>
The solution</h4>
The solution is fairly simple and uses DynamoDB's <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html#WorkingWithItems.ConditionalUpdate">Conditional Writes</a> for synchronisation and SQS <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-timers.html">Message Timers</a> to enable aggregation. When the Lambda function processes a new object event it first checks to see if the event falls within the window of the currently active refresh request. If the event is within the window it will automatically be included when the refresh executes and the event can be ignored. If the event occurred after the last refresh then a new refresh request is sent to an SQS queue with a message timer equal to the refresh window period. This allows for all messages received within a refresh window to be included in a single refresh operation.<br />
<br />
<h4>
Implementation</h4>
<div>
At a high level we need to create resources (SQS queue, DynamoDB table, Lambda functions), set up permissions (create and assign IAM roles), and apply some configuration (linking Lambda to S3 event notification and SQS queues). This implementation really belongs in a CloudFormation template (and I may actually create one) but I was interested to try and do this entirely via the AWS CLI, masochistic as that may be. If you are not interested in the gory implementation details then skip ahead to 'Creation and deletion script' section<br />
<br />
<script src="https://github.com/watchamcb/s3-event-aggregator/blob/f324ca33e18d0dcc7dd45e1294d0b2aeadb03b7b/iam/dynamo-writer.json#L1"> </script>
Let's start with the S3 event aggregation piece. We need:<br />
<ol>
<li>A DynamoDB table to track state</li>
<li>An SQS queue as a destination for aggregated actions</li>
<li>A Lambda function for processing and aggregating the S3 events</li>
<li>IAM permissions for all of the above</li>
</ol>
<div>
As the DynamoDB table and SQS queue are independent we can create these first:<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws dynamodb create-table --table-name S3EventAggregator --attribute-definitions AttributeName=BucketName,AttributeType=S </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">--key-schema AttributeName=BucketName,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=5</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws sqs create-queue --queue-name S3EventAggregatorActionQueue</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span><span style="font-family: inherit;">
Naturally this needs to be done with a user that has sufficient permissions and assumes your default region is set.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The Lambda function is a bit trickier as it requires a role to be created before the function can be created. So let's start with the IAM permissions. First let's create a policy allowing DynamoDB GetItem and UpdateItem to be performed on the DynamoDB table we created earlier. To do this we need a JSON file containing the necessary permissions. </span>The <a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/iam/dynamo-writer.json">dynamo-writer.json</a> file looks like this:<br />
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">{</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Version": "2012-10-17",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Statement": [</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> {</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Effect": "Allow",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Action": [</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><span style="white-space: pre;"> </span> "dynamodb:UpdateItem",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><span style="white-space: pre;"> </span> "dynamodb:GetItem"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><span style="white-space: pre;"> </span> ],</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Resource": "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/S3EventAggregator"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ]</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">}</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span></div>
<div>
We need to replace REGION and ACCOUNT_ID with the relevant values. As we are aiming at using the command line for this exercise, let's use STS to retrieve our account ID, set our region, and then use sed to substitute both variables:<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) </span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">AWS_DEFAULT_REGION="eu-west-1" </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O dynamo-writer.json </span><span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/iam/dynamo-writer.json </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/ACCOUNT_ID/$ACCOUNT_ID/g" dynamo-writer.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/REGION/$AWS_DEFAULT_REGION/g" dynamo-writer.json</span><br />
<div>
</div>
</div>
<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam create-policy --policy-name S3EventAggregatorDynamo --policy-document file://dynamo-writer.json</span><br />
<div>
<div>
<br /></div>
<div>
We now have a policy that allows the caller (our soon to be created Lambda function in this case) to update items in the S3EventAggregator DynamoDB table. Next we need to create a policy to allow the function to write messages to SQS. The <a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/iam/sqs-writer.json">sqs-writer.json</a> policy file contents are similar to the DynamoDB policy:<br />
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small; white-space: pre-wrap;">{</span><br />
<pre style="overflow-wrap: break-word; white-space: pre-wrap; word-wrap: break-word;"><span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:S3EventAggregatorActionQueue"
}
]
}</span></pre>
</div>
<div>
<br /></div>
<div>
Retrieving the file and substituting the ACCOUNT_ID and REGION using the environment variables we created for the DynamoDB policy:<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O sqs-writer.json </span><span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/iam/sqs-writer.json </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/ACCOUNT_ID/$ACCOUNT_ID/g" sqs-writer.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/REGION/$AWS_DEFAULT_REGION/g" sqs-writer.json</span></div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam create-policy --policy-name S3EventAggregatorSqsWriter --policy-document file://sqs-writer.json</span><br />
<br />
Having defined the two resource access policies let's create an IAM role for the Lambda function. </div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O lambda-trust.json </span><span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/iam/lambda-trust.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam create-role --role-name S3EventAggregatorLambdaRole --assume-role-policy-document file://lambda-trust.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span>
The <a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/iam/lambda-trust.json">lambda-trust.json</a> policy allows Lambda access to assume roles via STS and looks like this (no substitutions required for this one):<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">{</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Version": "2012-10-17",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Statement": [</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> {</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Effect": "Allow",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Principal": {</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Service": "lambda.amazonaws.com"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> },</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Action": "sts:AssumeRole"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ]</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">}</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span>
<span style="font-family: inherit;">We can now attach the SQS and DynamoDB policies to the new created Lambda role. We also need the AWS</span>LambdaBasicExecutionRole which is an AWS managed policy providing access to CloudWatch logs and Lambda function execution:<br />
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam attach-role-policy --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/S3EventAggregatorDynamo --role-name S3EventAggregatorLambdaRole</span></div>
</div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam attach-role-policy --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/S3EventAggregatorSqsWriter --role-name S3EventAggregatorLambdaRole</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole --role-name S3EventAggregatorLambdaRole</span></div>
</div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></span><br />
<div style="font-family: "Times New Roman"; font-size: medium;">
</div>
<span style="font-family: inherit;">We are now finally ready to create the </span><a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/src/s3_aggregator.py" style="font-family: inherit;">S3EventAggregator Lambda function</a><span style="font-family: inherit;">. Starting by creating the </span><a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html" style="font-family: inherit;">Python deployment package</a><span style="font-family: inherit;">:</span><br />
<div style="orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; widows: 2;">
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O s3_aggregator.py https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/src/s3_aggregator.py</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">zip function.zip s3_aggregator.py</span><br />
<br />
And then creating the function (using the AWS_DEFAULT_REGION and ACCOUNT_ID environment variables again):<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws lambda create-function --function-name S3EventAggregator --runtime python3.6 </span><span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">--role arn:aws:iam::$ACCOUNT_ID:role/S3EventAggregatorLambdaRole --zip-file fileb://function.zip </span><span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">--handler s3_aggregator.lambda_handler --timeout 10 </span><span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">--environment "Variables={QUEUE_URL=https://sqs.$AWS_DEFAULT_REGION.amazonaws.com/$ACCOUNT_ID/S3EventAggregatorActionQueue,REFRESH_DELAY_SECONDS=30,LOG_LEVEL=INFO}"</span><br />
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws lambda put-function-concurrency --function-name S3EventAggregator --reserved-concurrent-executions 1</span></div>
<div>
<br /></div>
<div>
The function concurrency is set to 1 as there is no benefit to having the function processing S3 events concurrently and 'single threading' the function will limit the maximum concurrent DynamoDB request rate to reduce DynamoDB capacity usage and costs.</div>
<div>
<br /></div>
<div>
All that is left now is to give S3 permission to execute the Lambda function and link the bucket notification events to the S3EventAggregator function. Giving S3 permission on the specific bucket:</div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">BUCKET=my-bucket</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws lambda add-permission --function-name S3EventAggregator --statement-id SID_$BUCKET --action lambda:InvokeFunction --principal s3.amazonaws.com --source-account $ACCOUNT_ID --source-arn arn:aws:s3:::$BUCKET</span></div>
<div>
<br /></div>
<div>
Interestingly, the --source-arn can be omitted to avoid needing to add permissions for each bucket you want the function to operate on but it is required (and must match a specific bucket) for the Lambda Console to display the function and trigger correctly. The S3 <a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/s3/event.json">event.json</a> configuration creates an event on any object creation or removal events:</div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">{</span></div>
<div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "LambdaFunctionConfigurations": [</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> {</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Id": "s3-event-aggregator",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "LambdaFunctionArn": "arn:aws:lambda:REGION:ACCOUNT_ID:function:S3EventAggregator",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Events": [</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "s3:ObjectCreated:*",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "s3:ObjectRemoved:*"</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ]</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ]</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">}</span></div>
</div>
<div>
<br /></div>
<div>
Once again substituting the relevant region and account IDs:</div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O event.json https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/s3/event.json</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/ACCOUNT_ID/$ACCOUNT_ID/g" event.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/REGION/$AWS_DEFAULT_REGION/g" event.json</span></div>
<div>
<br /></div>
<div>
And linking the event configuration to a bucket:</div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws s3api put-bucket-notification-configuration --bucket $BUCKET --notification-configuration file://event.json</span></div>
<div>
<br /></div>
<div>
Thus concludes the event aggregation part of the solution. A quick test confirms the event aggregation is working as expected:</div>
<div>
<br /></div>
<div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">time for i in $(seq 1 5); do aws s3 cp test.txt s3://$BUCKET/test$i.txt; done</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test1.txt </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test2.txt </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test3.txt </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test4.txt </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test5.txt </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">real<span style="white-space: pre;"> </span>0m2.106s</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">user<span style="white-space: pre;"> </span>0m1.227s</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sys<span style="white-space: pre;"> </span>0m0.140s</span></div>
</div>
<div>
<br /></div>
<div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">STREAM=$(aws logs describe-log-streams --log-group-name /aws/lambda/S3EventAggregator --order-by LastEventTime --descending --query 'logStreams[0].logStreamName' --output text); aws logs get-log-events --log-group-name /aws/lambda/S3EventAggregator --log-stream-name $STREAM --query 'events[*].{msg:message}' --output text | grep "^\[" | sed 's/\t/ /g'</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">[INFO] 2018-08-12T18:14:03.647Z 7da07415-9e5b-11e8-ab6d-8f962149ce24 Sending refresh request for bucket: s3-test-net, timestamp: 1534097642149</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">[INFO] 2018-08-12T18:14:04.207Z 7e1d6ca2-9e5b-11e8-ac9d-e1f0f9729f66 Refresh for bucket: s3-test-net within refresh window, skipping. S3 Event timestamp: 1534097642938</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">[INFO] 2018-08-12T18:14:04.426Z 7eefb013-9e5b-11e8-ab6d-8f962149ce24 Refresh for bucket: s3-test-net within refresh window, skipping. S3 Event timestamp: 1534097643812</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">[INFO] 2018-08-12T18:14:04.635Z 7e5b5feb-9e5b-11e8-8aa9-7908c99c450a Refresh for bucket: s3-test-net within refresh window, skipping. S3 Event timestamp: 1534097643371</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">[INFO] 2018-08-12T18:14:05.915Z 7ddb5a72-9e5b-11e8-80de-0dd6a15c3f62 Refresh for bucket: s3-test-net within refresh window, skipping. S3 Event timestamp: 1534097642517</span></div>
</div>
<div>
<br /></div>
<div>
From the 'within refresh window' log messages we can see 4 of the 5 events were skipped as they fell within the refresh aggregation window. Checking the SQS queue we can see the refresh request event:</div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws sqs receive-message --queue-url https://$AWS_DEFAULT_REGION.queue.amazonaws.com/$ACCOUNT_ID/S3EventAggregatorActionQueue --attribute-names All --message-attribute-names All</span></div>
<div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">{</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Messages": [</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> {</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "MessageId": "c0027dd2-30bc-48bc-b622-b5c85d862c92",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "ReceiptHandle": "AQEB9DQXkIWsWn...5XU2a13Q8=",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "MD5OfBody": "99914b932bd37a50b983c5e7c90ae93b",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Body": "{}",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Attributes": {</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "SenderId": "AROAI55PXBF63XVSEBNYM:S3EventAggregator",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "ApproximateFirstReceiveTimestamp": "1534097653846",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "ApproximateReceiveCount": "1",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "SentTimestamp": "1534097642728"</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> },</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "MD5OfMessageAttributes": "6f6eaf397811cbece985f3e8d87546c3",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "MessageAttributes": {</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "bucket-name": {</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "StringValue": "s3-test-net",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "DataType": "String"</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> },</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "timestamp": {</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "StringValue": "1534097642149",</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "DataType": "Number"</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ]</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">}</span></div>
</div>
<div>
<br /></div>
<div>
Moving onto the final part of the solution, we need a Lambda function that processes the events that the S3EventAggregator function sends to SQS. For the function's permissions we can reuse the S3EventAggregatorDynamo policy for DynamoDB access but will need to create a new policy for reading and deleting SQS messages and refreshing the Storage Gateway cache.<br />
<br />
The <a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/iam/sgw-refresh.json">sgw-refresh.json</a> is as follows, note that SMB file shares are included but the current <a href="https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html">Lambda execution environment</a> only supports boto3 1.7.30 which does not actually expose the SMB APIs (more on working around this later):<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">{</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Version": "2012-10-17",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Statement": [</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> {</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Effect": "Allow",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Action": [</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "storagegateway:RefreshCache",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "storagegateway:ListFileShares",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "storagegateway:DescribeNFSFileShares",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "storagegateway:DescribeSMBFileShares"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ],</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Resource": "*"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ]</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">}</span><br />
<br />
Creating the policy:<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O sgw-refresh.json https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/iam/sgw-refresh.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam create-policy --policy-name StorageGatewayRefreshPolicy --policy-document file://sgw-refresh.json</span><br />
<br />
The <a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/iam/sqs-reader.json">sqs-reader.json</a> gives the necessary SQS read permissions on the S3EventAggregatorActionQueue:<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">{</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Version": "2012-10-17",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Statement": [</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> {</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Effect": "Allow",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Action": [</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "sqs:DeleteMessage",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "sqs:GetQueueAttributes",</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "sqs:ReceiveMessage"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ],</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:S3EventAggregatorActionQueue"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> }</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"> ]</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">}</span><br />
<br />
Substituting and creating the policy:<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O sqs-reader.json https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/iam/sqs-reader.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/ACCOUNT_ID/$ACCOUNT_ID/g" sqs-reader.json </span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">sed -i "s/REGION/$AWS_DEFAULT_REGION/g" sqs-reader.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam create-policy --policy-name S3EventAggregatorSqsReader --policy-document file://sqs-reader.json</span><br />
<div>
<br /></div>
And then creating the role and adding the relevant policies:<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O lambda-trust.json https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/iam/lambda-trust.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam create-role --role-name S3AggregatorActionLambdaRole --assume-role-policy-document file://lambda-trust.json</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole --role-name S3AggregatorActionLambdaRole</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam attach-role-policy --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/S3EventAggregatorSqsReader --role-name S3AggregatorActionLambdaRole</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam attach-role-policy --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/S3EventAggregatorDynamo --role-name S3AggregatorActionLambdaRole</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws iam attach-role-policy --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/StorageGatewayRefreshPolicy --role-name S3AggregatorActionLambdaRole</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span>
Next we will create the Lambda function, this will however depend on whether or not you require SMB file share support. As mentioned earlier the current Lambda execution environment does not expose the new SMB file share APIs so if you have SMB shares mapped on your Storage Gateway you will have to <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html">include the latest botocore and boto3</a> libraries with your deployment. The disadvantage of this is that you are not able to view the code in the Lambda console (due to the deployment file size limitation). If you are only using NFS shares then you only need the code without the latest libraries but it will break if you add an SMB share before the Lambda execution environment supports it. Including the dependency in the deployment is the preferred option so that is what we are going to do:<br />
<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">mkdir deploy</span><br />
<div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">pip install boto3 botocore -t deploy</span></div>
</div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">wget -O deploy/s3_sgw_refresh.py https://raw.githubusercontent.com/watchamcb/s3-event-aggregator/master/src/s3_sgw_refresh.py</span></div>
<div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">cd deploy</span></div>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">zip -r function.zip *</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;"><br /></span>
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws lambda create-function --function-name S3StorageGatewayRefresh --runtime python3.6 --role arn:aws:iam::$ACCOUNT_ID:role/S3AggregatorActionLambdaRole --zip-file fileb://function.zip --handler s3_sgw_refresh.lambda_handler --timeout 5 </span><span face=""helvetica neue", arial, helvetica, sans-serif" style="font-size: x-small;">--environment "Variables={LOG_LEVEL=INFO}"</span><br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws lambda put-function-concurrency --function-name S3StorageGatewayRefresh --reserved-concurrent-executions 1</span><br />
<br />
And finally create the event mapping to execute the <a href="https://github.com/watchamcb/s3-event-aggregator/blob/master/src/s3_sgw_refresh.py">S3StorageGatewayRefresh</a> function when messages are received on the queue:<br />
<br />
<span face=""helvetica neue" , "arial" , "helvetica" , sans-serif" style="font-size: x-small;">aws lambda create-event-source-mapping --function-name S3StorageGatewayRefresh --event-source arn:aws:sqs:$AWS_DEFAULT_REGION:$ACCOUNT_ID:S3EventAggregatorActionQueue --batch-size 1</span><br />
<br />
And that is the final solution. Verifying it works as expected, let's mount the NFS share and upload some files via the CLI and confirm the share is refreshed. Mounting the share:<br />
<br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">$ sudo mount -t nfs -o nolock 172.31.2.13:/s3-test-net share</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">$ cd share/sgw</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">$ ls -l</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">total 0</span><br />
<div>
<br /></div>
<div>
Uploading the files through the CLI (from a different machine):</div>
<br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">$ for i in $(seq 1 20); do aws s3 cp test.txt s3://$BUCKET/sgw/test$i.txt; done</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test1.txt </span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test2.txt </span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">...</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test19.txt </span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">upload: ./test.txt to s3://s3-test-net/test20.txt </span><br />
<br />
And confirming the refresh of the share:<br />
<br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">$ ls -l</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">total 10</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">-rw-rw-rw- 1 nobody nogroup 19 Aug 25 07:46 test10.txt</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">-rw-rw-rw- 1 nobody nogroup 19 Aug 25 07:46 test11.txt</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">...</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">-rw-rw-rw- 1 nobody nogroup 19 Aug 25 07:46 test8.txt</span><br />
<span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-size: x-small;">-rw-rw-rw- 1 nobody nogroup 19 Aug 25 07:46 test9.txt</span></div>
<div>
<br /></div>
<div>
<h4>
Creation and deletion scripts</h4>
<div>
For convenience a script to create (and remove) this stack is provided on GitHub. Clone the <a href="https://github.com/watchamcb/s3-event-aggregator">s3-event-aggregator</a> repository and run the create-stack.sh and delete-stack.sh scripts respectively. You need to have the AWS CLI installed and configured and sed and zip must be available. Be sure to edit the BUCKET variable in the script to match your bucket name and change the REGION if appropriate.</div>
<div>
<br /></div>
<div>
Note that the delete stack script will not remove the S3 event notification configuration by default. There is no safe and convenient way to remove only the S3EventAggregator configuration (other than removing all configuration which may result in unintended loss of other event configuration). If you have other events configured on a bucket it is best to use the AWS Console to remove the s3-event-aggregator event configuration. If there are no other events configured on your bucket you can safely uncomment the relevant line in the deletion script.</div>
</div>
<div>
<br /></div>
<h4>
Configuration</h4>
<div>
The two Lambda functions both have a LOG_LEVEL environment variable to control the details logged to CloudWatch Logs, the functions were created with the level set to INFO but DEBUG may be useful for troubleshooting and WARN is probably appropriate for use in production.</div>
<div>
<br /></div>
<div>
The S3EventAggregator function also has an environment variable called REFRESH_DELAY_SECONDS for controlling the event aggregation window. It was initialised to 30 seconds when the function was created but it may be appropriate to change it depending on your S3 upload pattern. If the uploads are mostly small and complete quickly, or if you need the Storage Gateway to reflect changes quickly then this may be a reasonable value. If you are performing larger uploads or the total upload process takes significantly longer then the refresh window would need to be increased to be longer than the total expected upload time.</div>
<div>
<br /></div>
<div>
The DynamoDB table was created with 5 write capacity units and as the entries are less than 1KB this should be sufficient as long as you are not writing more than 5 objects a second to the Storage Gateway S3 bucket. Writing more than this will required additional write capacity to be provisioned (or auto scaling enabled).</div>
<div>
<br /></div>
<div>
The same code can be used for multiple buckets by simply adding additional bucket event configurations via the CLI put-bucket-notification-configuration as above or using the AWS Console.</div>
<div>
<br /></div>
<h4>
Cost</h4>
<div>
There are three component costs involved in this solution, the two Lambda functions, DynamoDB, and SQS. The Lambda and DynamoDB costs will scale fairly linearly with usage with both the S3EventAggregator and DynamoDB being charged for each S3 event that is triggered. To get an idea of the number of events to expect you can enable <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/metrics-configurations.html">S3 metrics</a> on the bucket and check the PUT and DELETE counts. The S3StorageGatewayRefresh function and SQS messages will be a fraction of the total S3 event counts and dependent on the REFRESH_DELAY_SECONDS configuration. A longer refresh delay will result in fewer SQS messages and S3StorageGatewayRefresh function executions.</div>
<div>
<br /></div>
<div>
As an example lets use an example of 1000 objects uploaded a day with these being aggregated into 50 refresh events. For simplicity we will also assume that the free tier has been exhausted and that there are 30 days in the month. The total Lambda request count will then be:</div>
<div>
(30 days x 1000 S3 events) + (30 days x 50 refresh events) = 31 500 request</div>
<div>
As Lambda requests are charged in 1 million increments this will result in a charge of $0.2 for the requests</div>
<div>
<br /></div>
<div>
The compute charges are based on duration, with the S3EventAggregator executing in less than 100ms for all aggregate events and around 300 - 600ms for the refresh events. The S3StorageGatewayRefesh function takes between 400ms and 800ms. Giving us:</div>
<div>
(30 days x 950 S3 requests x 0.1s) + (30 days x 50 S3 requests x 0.5s) + (30 days x 50 refresh events x 0.7s) = 4650 seconds</div>
<div>
Lambda compute is charged in GB-s, so:</div>
<div>
4650 seconds x 128MB/1024 = 581.25 GB-s at $0.00001667 = $0.0096894</div>
<div>
Bringing the total Lambda charges for the month to $0.21</div>
<div>
<br /></div>
<div>
For DynamoDB we have provisioned 5 WCU at $0.000735/WCU/hour and 1 RCU at $0.000147/RCU/hour working out as:</div>
<div>
(5 WCU x 0.000735 per hour x 24 hours a day x 30 days) + (1 RCU x 0.000147 per hour x 24 hours x 30 days) = $2.75 a month</div>
<div>
<br /></div>
<div>
SQS charges per million requests with the 1500 send message requests and a further 1500 receive and delete requests all falling under this limit (and thus only costing $0.40 for the month). It is worth noting that Lambda does poll the SQS queue roughly 4 times a minute and this will contribute to your total SQS request costs, using around 172,800 SQS requests a month.</div>
<div>
<br /></div>
<div>
There are some other costs associated with CloudWatch Logs and DynamoDB storage but these should be fairly small compared to the request costs and I would not expect the total cost of the stack to be more than $10 - $15 a month.</div>
<div>
<br /></div>
<h4>
Conclusion</h4>
<div>
And so ends this post, well done for reading to the end. I quite enjoyed building this solution and will look at converting it to a CloudFormation template at a later stage. Feel free to log issues or pull requests against the GitHub repo.</div>
</div>
</div>
</div>
</div>Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-73152451163279369652018-05-18T09:28:00.000+01:002018-05-18T09:28:46.412+01:00Working with 20+ node ElastiCache Memcached clustersElastiCache Memcached <a href="https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_elasticache">limits</a> the number of nodes in a cluster to 20 by default. This limit can be <a href="https://aws.amazon.com/contact-us/elasticache-node-limit-request/">increased</a> with ElastiCache recommending against clusters larger than 50 nodes.<br />
<br />
The increased cluster node limits don't reflect in the console so even after the limit has been increased the maximum number of nodes you can select when creating a new cluster is 20. Fortunately the CLI allows you to work around this and you can tweak the examples in the ElastiCache cluster creation documentation to achieve your desired outcome. Creating a cluster with more than 20 nodes using the <a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/Clusters.Create.CLI.html">CLI</a>:<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ aws elasticache create-cache-cluster --cache-cluster-id my-super-cluster --cache-node-type cache.t2.micro --engine memcached --engine-version 1.4.34 --cache-parameter-group default.memcached1.4 --num-cache-nodes 30</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">{</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheCluster": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterId": "my-super-cluster",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheNodeType": "cache.t2.micro",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Engine": "memcached",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "EngineVersion": "1.4.34",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterStatus": "creating",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "NumCacheNodes": 30,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PreferredMaintenanceWindow": "mon:02:30-mon:03:30",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PendingModifiedValues": {},</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheSecurityGroups": [],</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheParameterGroup": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheParameterGroupName": "default.memcached1.4",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ParameterApplyStatus": "in-sync",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheNodeIdsToReboot": []</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheSubnetGroupName": "default",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "AutoMinorVersionUpgrade": true,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "TransitEncryptionEnabled": false,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "AtRestEncryptionEnabled": false</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">}</span><br />
<div>
<br /></div>
<div>
Adding nodes can be done via the console as it allows you to specify the number of nodes rather than selecting from a drop down. For completeness this can also be done via the <a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/Clusters.AddNode.html#Clusters.AddNode.CLI">CLI</a>:</div>
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ aws elasticache modify-cache-cluster --cache-cluster-id test-limits --num-cache-nodes 26 --apply-immediately</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">{</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheCluster": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterId": "test-limits",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ConfigurationEndpoint": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Address": "test-limits.rftr8g.cfg.euw1.cache.amazonaws.com",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Port": 11211</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheNodeType": "cache.t2.micro",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Engine": "memcached",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "EngineVersion": "1.4.34",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterStatus": "modifying",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "NumCacheNodes": 6,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PreferredAvailabilityZone": "Multiple",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterCreateTime": "2018-05-18T07:10:22.530Z",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PreferredMaintenanceWindow": "sat:23:30-sun:00:30",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PendingModifiedValues": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "NumCacheNodes": 26</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheSecurityGroups": [],</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheParameterGroup": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheParameterGroupName": "default.memcached1.4",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ParameterApplyStatus": "in-sync",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheNodeIdsToReboot": []</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheSubnetGroupName": "default",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "AutoMinorVersionUpgrade": true,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "SecurityGroups": [</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "SecurityGroupId": "sg-5814fd37",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Status": "active"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ],</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "TransitEncryptionEnabled": false,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "AtRestEncryptionEnabled": false</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">}</span><br />
<div>
<br /></div>
<br />
And finally <a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/Clusters.DeleteNode.html#Clusters.DeleteNode.CLI">removing nodes</a> via the CLI (note the spaces rather than commas between the node IDs):<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ aws elasticache modify-cache-cluster --cache-cluster-id test-limits --num-cache-nodes 6 --cache-node-ids-to-remove 0007 0008 0009 0010 0011 0012 0013 0014 0015 0016 0017 0018 0019 0020 0021 0022 0023 0024 0025 0026 --apply-immediately </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">{</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheCluster": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterId": "test-limits",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ConfigurationEndpoint": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Address": "test-limits.rftr8g.cfg.euw1.cache.amazonaws.com",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Port": 11211</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheNodeType": "cache.t2.micro",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Engine": "memcached",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "EngineVersion": "1.4.34",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterStatus": "modifying",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "NumCacheNodes": 26,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PreferredAvailabilityZone": "Multiple",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheClusterCreateTime": "2018-05-18T07:10:22.530Z",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PreferredMaintenanceWindow": "sat:23:30-sun:00:30",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "PendingModifiedValues": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "NumCacheNodes": 6,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheNodeIdsToRemove": [</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0007",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0008",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0009",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0010",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0011",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0012",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0013",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0014",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0015",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0016",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0017",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0018",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0019",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0020",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0021",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0022",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0023",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0024",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0025",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "0026"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ]</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheSecurityGroups": [],</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheParameterGroup": {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheParameterGroupName": "default.memcached1.4",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "ParameterApplyStatus": "in-sync",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheNodeIdsToReboot": []</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "CacheSubnetGroupName": "default",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "AutoMinorVersionUpgrade": true,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "SecurityGroups": [</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "SecurityGroupId": "sg-5814fd37",</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "Status": "active"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ],</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "TransitEncryptionEnabled": false,</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "AtRestEncryptionEnabled": false</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">}</span><br />
<div>
<br /></div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-57498579632366904072017-11-16T15:20:00.001+00:002017-11-16T16:21:31.806+00:00Extracting S3 bucket sizes using the AWS CLIA quick one liner for printing out the size (in bytes) of S3 StandardStorage buckets in your account (using bash):<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">for name in $(aws s3api list-buckets --query 'Buckets[*].Name' --output text); do size=$(aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --start-time $(date --date="yesterday" +%Y-%m-%d) --end-time $(date +%Y-%m-%d) --period 86400 --statistics Maximum --dimensions Name=BucketName,Value=$name Name=StorageType,Value=StandardStorage --query 'Datapoints[0].Maximum' | sed 's/null/0.0/' | cut -d. -f1); echo "$name,$size"; done</span><br />
<div>
<br /></div>
<div>
<br /></div>
<div>
Some of the individual components may be independently useful, starting off with listing buckets:</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws s3api list-buckets --query 'Buckets[*].Name' --output text</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<span style="font-family: inherit;">Pretty straight forward, uses the <a href="http://docs.aws.amazon.com/cli/latest/reference/s3api/index.html">s3api</a> in the CLI to list buckets returning only their names in text format.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
To get the bucket sizes we are actually querying CloudWatch (using <a href="http://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html">get-metric-statistics</a>) which provides bucket size and object count <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/s3-metricscollected.html">metrics for S3</a>:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --start-time $(date --date="yesterday" +%Y-%m-%d) --end-time $(date +%Y-%m-%d) --period 86400 --statistics Maximum --dimensions Name=BucketName,Value=$name Name=StorageType,Value=StandardStorage --query 'Datapoints[0].Maximum' | sed 's/null/0.0/' | cut -d. -f1</span><br />
<br />
Most of this is fairly straight forward some of the parameters worth explaining further:<br />
--start-time: we are setting the start date to yesterday and formatting it in yyyy-mm-dd format, this ensures we get at least one data point containing the bucket size<br />
--period: 86400 = 1 day as the bucket size metric is only published once a day (at 00:00:00)<br />
--query: we are only interested in the the actual value for one of the metric datapoints<br />
<br />
The sed and cut commands are just to clean up formatting, if the bucket is empty the CloudWatch metric request will return null so we replace null with 0.0. The bucket size metric value will always end in .0 so we truncate it using cut (yes there are other ways of doing this).<br />
<br />
If you are only wanting to check the size of a single bucket or the bucket size on a specific date you can use a simpler version:<br />
<br class="Apple-interchange-newline" />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --start-time 2017-11-15 --end-time 2017-11-16 --period 86400 --statistics Maximum --dimensions Name=BucketName,Value=MY_BUCKET_NAME Name=StorageType,Value=StandardStorage --query 'Datapoints[0].Maximum'</span>Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-18573597699051702422017-08-20T15:36:00.004+01:002017-08-21T07:24:57.329+01:00Understanding EC2 "Up to 10 Gigabit" network performance for R4 instances<span style="background-color: white; font-size: 16px;">This post investigates the network performance of AWS R4 instances with a focus on the "Up to 10 Gigabit" networking expected from smaller (r4.large - r4.4xlarge) instance types. Before starting it should be noted that this post is based on observation and as such is prone to imprecision and variance, it is intended as a guide for what can be expected and not a comprehensive or scientific review.<br />
</span><br />
<span style="background-color: white; font-size: 16px;"><br /></span><span style="background-color: white; font-size: 16px;">
The R4 instance <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/memory-optimized-instances.html#memory-instances-hardware">documentation</a> states "The smaller R4 instance sizes offer peak throughput of 10 Gbps. These instances use a network I/O credit mechanism to allocate network bandwidth to instances based on average bandwidth utilization. These instances accrue credits when their network throughput is below their baseline limits, and can use these credits when they perform network data transfers." This is not particularly helpful in understanding the lower bounds on network performance and gives no indication of the baseline limits with AWS recommending customers <a href="https://aws.amazon.com/premiumsupport/knowledge-center/network-throughput-benchmark-linux-ec2/">benchmark</a> the networking performance of various instances to evaluate whether the instance type and size will meet the application network performance requirements.</span><span style="background-color: white; font-size: 16px;"><br /></span><span style="background-color: white; font-size: 16px;">
Logically we would expect the r4.large to have a fraction of the total 20 Gbps available on an r4.16xlarge. From the instance size normalisation table under the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-modifying.html#ri-modification-instancemove">reserved instance modification documentation</a> a *.large instance (factor of 4) should expect 1/32 of the resources available on a *.16xlarge instance (factor of 128) which works out at 0.625 Gbps (20 Gbps / 32) or 625 Mbps.</span><br />
<br />
<h4>
<br class="Apple-interchange-newline" /><span style="background-color: white; font-size: 16px;">Testing r4.large baseline network performance</span></h4>
<div>
<span style="background-color: white; font-size: 16px;">
Using iperf3 between two newly launched Amazon Linux r4.large instances in the same availability zone in eu-west-1, we run into the first interesting anomaly with the network stream maxing out at 5 Gbps rather than the expected 10 Gbps:<br />
</span>
<br />
<span style="background-color: white; font-size: 16px;"><br /></span></div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ iperf3 -p 5201 -c 172.31.7.67 -i 1 -t 3600 -f m -V </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">iperf 3-CURRENT</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Linux ip-172-31-10-235 4.9.32-15.41.amzn1.x86_64 #1 SMP Thu Jun 22 06:20:54 UTC 2017 x86_64</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Control connection MSS 8949</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Time: Sun, 20 Aug 2017 07:35:48 GMT</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Connecting to host 172.31.7.67, port 5201</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Cookie: p2v6ry2kzjo2udittrzgmxotz7we3in5etmv</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> TCP MSS: 8949 (default)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] local 172.31.10.235 port 41270 connected to 172.31.7.67 port 5201</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 3600 second test, tos 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ ID] Interval Transfer Bitrate Retr Cwnd</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 0.00-1.00 sec 598 MBytes 5015 Mbits/sec 9 664 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 1.00-2.00 sec 596 MBytes 4999 Mbits/sec 3 559 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 2.00-3.00 sec 595 MBytes 4992 Mbits/sec 9 586 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 3.00-4.00 sec 595 MBytes 4989 Mbits/sec 0 638 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 4.00-5.00 sec 596 MBytes 5000 Mbits/sec 0 638 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 5.00-6.00 sec 595 MBytes 4989 Mbits/sec 0 638 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 6.00-7.00 sec 595 MBytes 4990 Mbits/sec 6 638 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 7.00-8.00 sec 595 MBytes 4990 Mbits/sec 3 524 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 8.00-9.00 sec 596 MBytes 4997 Mbits/sec 0 586 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 9.00-10.00 sec 596 MBytes 4997 Mbits/sec 0 603 KBytes </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 10.00-11.00 sec 595 MBytes 4990 Mbits/sec 0 638 KBytes </span><br />
<div>
<br /></div>
<div>
Interestingly, using 2 parallel streams results in us (mostly) reaching the advertised 10 Gbps:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ iperf3 -p 5201 -c 172.31.7.67 -i 1 -t 3600 -f m -V -P 2</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">iperf 3-CURRENT</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Linux ip-172-31-10-235 4.9.32-15.41.amzn1.x86_64 #1 SMP Thu Jun 22 06:20:54 UTC 2017 x86_64</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Control connection MSS 8949</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Time: Sun, 20 Aug 2017 07:37:38 GMT</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Connecting to host 172.31.7.67, port 5201</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Cookie: q343avscwpva5uyg2ayeinboxi5pllvw5l7r</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> TCP MSS: 8949 (default)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] local 172.31.10.235 port 41274 connected to 172.31.7.67 port 5201</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] local 172.31.10.235 port 41276 connected to 172.31.7.67 port 5201</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Starting Test: protocol: TCP, 2 streams, 131072 byte blocks, omitting 0 seconds, 3600 second test, tos 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ ID] Interval Transfer Bitrate Retr Cwnd</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 0.00-1.00 sec 597 MBytes 5010 Mbits/sec 0 690 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 0.00-1.00 sec 592 MBytes 4968 Mbits/sec 0 717 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 0.00-1.00 sec 1.16 GBytes 9979 Mbits/sec 0 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 1.00-2.00 sec 595 MBytes 4994 Mbits/sec 0 690 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 1.00-2.00 sec 592 MBytes 4962 Mbits/sec 18 638 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 1.00-2.00 sec 1.16 GBytes 9956 Mbits/sec 18 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 2.00-3.00 sec 591 MBytes 4957 Mbits/sec 137 463 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 2.00-3.00 sec 587 MBytes 4924 Mbits/sec 41 725 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 2.00-3.00 sec 1.15 GBytes 9881 Mbits/sec 178 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 3.00-4.00 sec 593 MBytes 4973 Mbits/sec 46 367 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 3.00-4.00 sec 591 MBytes 4956 Mbits/sec 40 419 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 3.00-4.00 sec 1.16 GBytes 9929 Mbits/sec 86 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 4.00-5.00 sec 592 MBytes 4968 Mbits/sec 141 542 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 4.00-5.00 sec 591 MBytes 4960 Mbits/sec 36 559 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 4.00-5.00 sec 1.16 GBytes 9928 Mbits/sec 177 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 5.00-6.00 sec 595 MBytes 4995 Mbits/sec 30 664 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 5.00-6.00 sec 588 MBytes 4934 Mbits/sec 8 568 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 5.00-6.00 sec 1.16 GBytes 9929 Mbits/sec 38 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 6.00-7.00 sec 596 MBytes 5000 Mbits/sec 0 664 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 6.00-7.00 sec 589 MBytes 4945 Mbits/sec 0 629 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 6.00-7.00 sec 1.16 GBytes 9945 Mbits/sec 0 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 7.00-8.00 sec 588 MBytes 4935 Mbits/sec 7 655 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 7.00-8.00 sec 594 MBytes 4982 Mbits/sec 0 682 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 7.00-8.00 sec 1.15 GBytes 9917 Mbits/sec 7 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 8.00-9.00 sec 593 MBytes 4974 Mbits/sec 8 620 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 8.00-9.00 sec 593 MBytes 4978 Mbits/sec 12 717 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 8.00-9.00 sec 1.16 GBytes 9952 Mbits/sec 20 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 5] 9.00-10.00 sec 596 MBytes 4999 Mbits/sec 0 638 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 7] 9.00-10.00 sec 590 MBytes 4951 Mbits/sec 0 717 KBytes </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 9.00-10.00 sec 1.16 GBytes 9950 Mbits/sec 0 </span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<br />
<span style="font-family: inherit;">This behaviour is not consistent, stopping and restarting the instances often resulted in the full 10 Gbps on a single stream suggesting the issue relates to instance placement, something that appears to be supported by the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#concepts-placement-groups">placement group documentation</a> which states: "Network traffic to and from resources outside the placement group is limited to 5 Gbps." It is also possible that the streams are incorrectly being treated as placement group or public internet flows with different limits. For consistency I have used two parallel streams to avoid this issue in the rest of the article.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The CloudWatch graphs shows us reaching a steady baseline after around eight minutes of starting iperf3:</span><br />
<div>
<br /></div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZ4vNHMtQp9j2PJbEl2ZmZAtx0HheYSQJDvNv0uuxTlTEQKsL9yUmMLcCZFXmBy5Znkw4I1YchF0tqEEyFzM4jSIsW8Y5ut0za4iu26OH7TqnaBa2SVw1WwmOs4uU1QDl3bzQZoHi35sA/s1600/r4base.png" imageanchor="1"><img border="0" data-original-height="419" data-original-width="885" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZ4vNHMtQp9j2PJbEl2ZmZAtx0HheYSQJDvNv0uuxTlTEQKsL9yUmMLcCZFXmBy5Znkw4I1YchF0tqEEyFzM4jSIsW8Y5ut0za4iu26OH7TqnaBa2SVw1WwmOs4uU1QDl3bzQZoHi35sA/s640/r4base.png" width="640" /></a></div>
<div>
<span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;">A quick word on the graph above, firstly it is in bytes and, having</span><span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;"> enabled detailed monitoring,</span><span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;"> one minute granularity. For conversion purpose this means we need to divide the value of the metric by 60 to get bytes per second and then multiple by 8 to get bits per seconds. Looking at the actual data from the graph above:</span><br />
<span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ aws cloudwatch get-metric-statistics --metric-name NetworkOut --start-time 2017-08-20T08:23:00 --end-time 2017-08-20T08:35:00 --period 60 --namespace AWS/EC2 --statistics Average --dimensions Name=InstanceId,Value=i-0a7e009e7c0bf8fa8 --query 'Datapoints[*].[Timestamp,Average]' --output=text | sort</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:23:00Z 486.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:24:00Z 5726.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:25:00Z 22711496136.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:26:00Z 76376122845.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:27:00Z 76403033046.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:28:00Z 76357957564.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:29:00Z 76304994405.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:30:00Z 48667898310.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:31:00Z 5776989873.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:32:00Z 5816890095.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:33:00Z 5692555065.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">2017-08-20T08:34:00Z 5692014471.0</span><br />
<span style="font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial";"><br /></span>
<span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;"><br /></span>
<span style="background-color: white; font-family: "open sans" , "lucida grande" , "helvetica neue" , "arial"; font-size: 16px;">The maximum average throughput (between 08:26 and 08:29) is around 76 GByte/minute which works out at around 1.2 GByte/second or approximately 10.1 Gbit/second. Similarly the baseline (from 08:31 onwards) is in the region of 5.7 GByte/minute which translates to around 94 MByte/second or around 750 Mbit/second. These numbers are naturally averages but are fairly close to the actual iperf3 results, with the peak throughput of just over 10Gbit/second:</span></div>
<div>
<span style="background-color: white;">
</span>
<br />
<div style="font-family: "Open Sans", "Lucida Grande", "Helvetica Neue", Arial; font-size: 16px;">
<span style="background-color: white;"><br /></span></div>
<span style="background-color: white;">
</span>
<div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ ID] Interval Transfer Bitrate Retr Cwnd</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 0.00-1.00 sec 604 MBytes 5065 Mbits/sec 0 551 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 0.00-1.00 sec 604 MBytes 5062 Mbits/sec 0 524 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 0.00-1.00 sec 1.18 GBytes 10127 Mbits/sec 0 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 1.00-2.00 sec 601 MBytes 5046 Mbits/sec 0 551 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 1.00-2.00 sec 602 MBytes 5048 Mbits/sec 0 551 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 1.00-2.00 sec 1.18 GBytes 10094 Mbits/sec 0 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 2.00-3.00 sec 602 MBytes 5046 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 2.00-3.00 sec 602 MBytes 5047 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 2.00-3.00 sec 1.17 GBytes 10093 Mbits/sec 0 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 3.00-4.00 sec 601 MBytes 5045 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 3.00-4.00 sec 602 MBytes 5049 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 3.00-4.00 sec 1.18 GBytes 10095 Mbits/sec 0 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 4.00-5.00 sec 602 MBytes 5049 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 4.00-5.00 sec 601 MBytes 5045 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 4.00-5.00 sec 1.18 GBytes 10094 Mbits/sec 0 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 5.00-6.00 sec 602 MBytes 5049 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 5.00-6.00 sec 601 MBytes 5042 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 5.00-6.00 sec 1.17 GBytes 10092 Mbits/sec 0 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 6.00-7.00 sec 602 MBytes 5046 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 6.00-7.00 sec 602 MBytes 5049 Mbits/sec 0 577 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 6.00-7.00 sec 1.18 GBytes 10095 Mbits/sec 0 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 7.00-8.00 sec 603 MBytes 5063 Mbits/sec 0 1.44 MBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 7.00-8.00 sec 601 MBytes 5041 Mbits/sec 66 524 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 7.00-8.00 sec 1.18 GBytes 10104 Mbits/sec 66 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div style="font-family: "Open Sans", "Lucida Grande", "Helvetica Neue", Arial; font-size: 16px;">
<span style="background-color: white;"><br /></span></div>
<div style="font-family: "Open Sans", "Lucida Grande", "Helvetica Neue", Arial; font-size: 16px;">
<span style="background-color: white;">And the baseline of around 750Mbit/second:</span></div>
<div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ ID] Interval Transfer Bitrate Retr Cwnd</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 376.00-377.00 sec 43.8 MBytes 367 Mbits/sec 157 114 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 376.00-377.00 sec 43.8 MBytes 367 Mbits/sec 157 78.7 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 376.00-377.00 sec 87.5 MBytes 734 Mbits/sec 314 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 377.00-378.00 sec 45.0 MBytes 377 Mbits/sec 161 69.9 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 377.00-378.00 sec 45.0 MBytes 377 Mbits/sec 167 78.7 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 377.00-378.00 sec 90.0 MBytes 755 Mbits/sec 328 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 378.00-379.00 sec 43.8 MBytes 367 Mbits/sec 182 69.9 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 378.00-379.00 sec 45.0 MBytes 377 Mbits/sec 168 105 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 378.00-379.00 sec 88.8 MBytes 744 Mbits/sec 350 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 379.00-380.00 sec 42.5 MBytes 357 Mbits/sec 150 61.2 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 379.00-380.00 sec 46.2 MBytes 388 Mbits/sec 165 96.1 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 379.00-380.00 sec 88.8 MBytes 744 Mbits/sec 315 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 380.00-381.00 sec 36.2 MBytes 304 Mbits/sec 129 78.7 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 380.00-381.00 sec 52.5 MBytes 440 Mbits/sec 203 105 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 380.00-381.00 sec 88.8 MBytes 744 Mbits/sec 332 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 381.00-382.00 sec 36.2 MBytes 304 Mbits/sec 147 96.1 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 381.00-382.00 sec 52.5 MBytes 440 Mbits/sec 220 87.4 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 381.00-382.00 sec 88.8 MBytes 744 Mbits/sec 367 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 382.00-383.00 sec 46.2 MBytes 388 Mbits/sec 175 52.4 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 382.00-383.00 sec 42.5 MBytes 357 Mbits/sec 167 114 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 382.00-383.00 sec 88.8 MBytes 744 Mbits/sec 342 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 383.00-384.00 sec 41.2 MBytes 346 Mbits/sec 165 61.2 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 383.00-384.00 sec 47.5 MBytes 398 Mbits/sec 170 96.1 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 383.00-384.00 sec 88.8 MBytes 744 Mbits/sec 335 </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 6] 384.00-385.00 sec 50.0 MBytes 419 Mbits/sec 195 87.4 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[ 8] 384.00-385.00 sec 38.8 MBytes 325 Mbits/sec 157 52.4 KBytes </span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[SUM] 384.00-385.00 sec 88.8 MBytes 744 Mbits/sec 352 </span></span></div>
<div style="font-size: 16px;">
<div style="font-family: "Times New Roman"; font-size: medium;">
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></span></div>
<h4>
<span style="background-color: white;">
</span></h4>
<h4>
<span style="background-color: white;">
Calculating network credit rates</span></h4>
<div>
<span style="background-color: white;">Using the baseline for network performance we can draw some inferences about the rate at which network credits are accrued. For simplicity I am going to define a network credit as having a value of 1 Gbps, so an instance with 10 network credits could transmit at 10 Gbps for 1 second, naturally if the instance network limit is 10 Gbps the maximum rate can't be exceeded even if the instance credit balance is sufficient (20 credits allows 2 seconds at 10 Gbps rather than 1 second at 20 Gbps) . Given the base network performance in the previous section, we can assume that an r4.large has a network credit rate of around 0.75 credits per second. We can also assume a starting balance of around 2700 as we were able to maintain 10 Gbps for around 295 seconds ((10 - 0.75) * 295) at the start of the iperf3 run. Finally it appears the maximum credit balance on the r4.large is the same as the initial balance. Leaving the instances idle for 3 hours should have resulted in a credit balance of around 8100 (0.75 rate * 3600 seconds in an hour * 3 hours) which should have theoretically allowed 810 seconds at 10 Gbps but instead provided only around 295 seconds.</span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<h4>
<span style="background-color: white;">
</span></h4>
<h4>
<span style="background-color: white;">
R4 network performance table</span></h4>
<div>
<span style="background-color: white;">Below is a table of the expected performance for R4 instance sizes.</span><br />
<span style="background-color: white;"><br /></span></div>
<table border="2" cellpadding="1" cellspacing="1" style="width: 100%px;">
<tbody>
<tr>
<th>Instance</th>
<th>Baseline Gbps (approximate)</th>
<th>Initial/Max Credit (approximate)</th>
<th>Maximum time<br />
at 10Gbps <br />
(approximate seconds)</th>
</tr>
<tr>
<td nowrap="">r4.large</td>
<td>0.75</td>
<td>2700</td>
<td>295</td>
</tr>
<tr>
<td nowrap="">r4.xlarge</td>
<td>1.25</td>
<td>5145</td>
<td>589</td>
</tr>
<tr>
<td nowrap="">r4.2xlarge</td>
<td>2.5</td>
<td>8925</td>
<td>1191</td>
</tr>
<tr>
<td nowrap="">r4.4xlarge</td>
<td>5</td>
<td>11950</td>
<td>2390</td>
</tr>
</tbody></table>
</div>
</div>
</div>
<span style="background-color: white;">
<div style="font-size: 16px;">
<br />
To calculate whether or not an instance will match your network throughput requirements take the difference from the baseline rate and your application base network utilisation and divide by the required burst rate.</div>
<div style="font-size: 16px;">
<br /></div>
<div style="font-size: 16px;">
For example, if your application requires a baseline of 0.6 Gbps, you would accrue credits at around 0.15 per second allowing you to burst for 10 Gbps for approximately one second every 66 seconds (10 / 0.15) or for 10 seconds every 660 seconds.</div>
</span></div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-40826960242477746092016-10-21T11:33:00.001+01:002016-10-21T11:33:33.017+01:00AWS CLI - Switching to and from regional EC2 reserved instancesAWS recently announced the availability of <a href="https://aws.amazon.com/blogs/aws/ec2-reserved-instance-update-convertible-ris-and-regional-benefit/">regional reserved</a> instances, this post explains how to switch a reservation from AZ specific to regional (and back) using the AWS CLI.<br />
<br />
<h4>
Step 1, find the reservation to modify</h4>
<span style="font-size: x-small;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">$ aws ec2 describe-reserved-instances --filters Name=state,Values=active</span></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">{</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstances": [</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesId": "c416aeaf-fb64-4218-970f-7426f6f32377", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "OfferingType": "No Upfront", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "AvailabilityZone": "eu-west-1c", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "End": "2017-10-21T08:45:55.000Z", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ProductDescription": "Linux/UNIX", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Scope": "Availability Zone", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "UsagePrice": 0.0, </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "RecurringCharges": [</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Amount": 0.01, </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Frequency": "Hourly"</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ], </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "OfferingClass": "standard", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Start": "2016-10-21T08:45:56.708Z", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "State": "active", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "FixedPrice": 0.0, </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "CurrencyCode": "USD", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Duration": 31536000, </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceTenancy": "default", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceType": "t2.micro", </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceCount": 1</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ]</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">}</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
The "Scope" field in the response shows that this reservation is currently specific to an Availability Zone, eu-west-1c in this case.<br />
<div>
<br /></div>
<h4>
Step 2, request the modification</h4>
<div>
<span style="font-size: x-small;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">$ aws ec2 modify-reserved-instances --reserved-instances-ids c416aeaf-fb64-4218-970f-7426f6f32377 --target-configurations Scope=Region,InstanceCount=1</span></span></div>
<div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">{</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesModificationId": "rimod-aaada6ed-fec9-47c7-92e2-6edf7e61f2ce"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">}</span></div>
</div>
<div>
<br /></div>
<div>
The Scope=Region indicates that this reservation should be converted to a regional reservation, InstanceCount is a required parameter to indicate the number of reservations the modification should be applied to.</div>
<div>
<br /></div>
<h4>
Step 3, monitor progress</h4>
<div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">$ aws ec2 describe-reserved-instances-modifications</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">{</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesModifications": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Status": "processing", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ModificationResults": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesId": "35f9b908-ae36-41ca-ac0b-4c67c887135b", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "TargetConfiguration": {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceCount": 1</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ], </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "EffectiveDate": "2016-10-21T08:45:57.000Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "CreateDate": "2016-10-21T08:50:28.585Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "UpdateDate": "2016-10-21T08:50:31.098Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesModificationId": "rimod-aaada6ed-fec9-47c7-92e2-6edf7e61f2ce", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesIds": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesId": "c416aeaf-fb64-4218-970f-7426f6f32377"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ]</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ]</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">}</span></div>
</div>
<div>
<br /></div>
<div>
The "Status" in the response will show "processing" until the modification has completed successfully, at which time it will change to "fulfilled":</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">$ aws ec2 describe-reserved-instances-modifications</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">{</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesModifications": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Status": "fulfilled", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ModificationResults": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesId": "35f9b908-ae36-41ca-ac0b-4c67c887135b", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "TargetConfiguration": {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceCount": 1</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ], </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "EffectiveDate": "2016-10-21T08:45:57.000Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "CreateDate": "2016-10-21T08:50:28.585Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "UpdateDate": "2016-10-21T09:11:33.454Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesModificationId": "rimod-aaada6ed-fec9-47c7-92e2-6edf7e61f2ce", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesIds": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesId": "c416aeaf-fb64-4218-970f-7426f6f32377"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ]</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ]</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">}</span></div>
</div>
<div>
<br /></div>
<h4>
Step 4, success!</h4>
<div>
The new reservation is now regional (Scope=Region):</div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">$ aws ec2 describe-reserved-instances --filters Name=state,Values=active</span><br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">{</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstances": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesId": "35f9b908-ae36-41ca-ac0b-4c67c887135b", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "OfferingType": "No Upfront", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "FixedPrice": 0.0, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "End": "2017-10-21T08:45:55.000Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ProductDescription": "Linux/UNIX", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Scope": "Region", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "UsagePrice": 0.0, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "RecurringCharges": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Amount": 0.01, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Frequency": "Hourly"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ], </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "OfferingClass": "standard", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Start": "2016-10-21T08:45:57.000Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "State": "active", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceCount": 1, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "CurrencyCode": "USD", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Duration": 31536000, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceTenancy": "default", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceType": "t2.micro"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ]</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">}</span></div>
</div>
<div>
<br /></div>
<h4>
Switching back</h4>
<div>
Follows the same process with the added requirement of specifying which AZ the reservation should be linked to:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">$ aws ec2 modify-reserved-instances --reserved-instances-ids 35f9b908-ae36-41ca-ac0b-4c67c887135b --target-configurations Scope="Availability Zone",InstanceCount=1,AvailabilityZone=eu-west-1b</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">{</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesModificationId": "rimod-9e490be9-55a3-48cf-81e9-2662b13db2f8"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">}</span></div>
</div>
<div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">$ aws ec2 describe-reserved-instances --filters Name=state,Values=active</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">{</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstances": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ReservedInstancesId": "df70d097-2f33-4962-bca6-37af15ca819e", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "OfferingType": "No Upfront", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "AvailabilityZone": "eu-west-1b", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "End": "2017-10-21T08:45:55.000Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "ProductDescription": "Linux/UNIX", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Scope": "Availability Zone", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "UsagePrice": 0.0, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "RecurringCharges": [</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> {</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Amount": 0.01, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Frequency": "Hourly"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ], </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "OfferingClass": "standard", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Start": "2016-10-21T08:45:58.000Z", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "State": "active", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "FixedPrice": 0.0, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "CurrencyCode": "USD", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "Duration": 31536000, </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceTenancy": "default", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceType": "t2.micro", </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> "InstanceCount": 1</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"> ]</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">}</span></div>
</div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-50961214176267323072016-08-31T07:25:00.003+01:002017-11-16T14:48:03.467+00:00AWS troubleshooting - Lamba deployment package file permissionsWhen creating your own <a href="http://docs.aws.amazon.com/lambda/latest/dg/deployment-package-v2.html">Lambda deployment packages</a> be aware of the permissions on the files before zipping them. Lambda requires the files to have read access for all users, particularly "other", if this is missing you will receive a non-obvious error when trying to call the function. The fix is simple enough, perform a 'chmod a+r *' before creating your zip file. If the code is visible in the inline editor adding an empty line and saving will also fix the problem, presumably by overwriting the file with the correct permissions.<br />
<div>
<br /></div>
<div>
Below are some examples of errors you will see in the various languages if read permissions are missing. Hopefully this post will have saved you some time debugging.</div>
<div>
<br /></div>
<div>
<b>Java CloudWatch logs:</b></div>
<div>
--</div>
<div>
<div>
Class not found: example.Hello: class java.lang.ClassNotFoundException</div>
<div>
java.lang.ClassNotFoundException: example.Hello</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.net.URLClassLoader$1.run(URLClassLoader.java:370)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.net.URLClassLoader$1.run(URLClassLoader.java:362)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.security.AccessController.doPrivileged(Native Method)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.net.URLClassLoader.findClass(URLClassLoader.java:361)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.lang.ClassLoader.loadClass(ClassLoader.java:424)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.lang.ClassLoader.loadClass(ClassLoader.java:357)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.lang.Class.forName0(Native Method)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.lang.Class.forName(Class.java:348)</div>
<div>
Caused by: java.io.FileNotFoundException: /var/task/example/Hello.class (Permission denied)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.FileInputStream.open0(Native Method)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.FileInputStream.open(FileInputStream.java:195)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.FileInputStream.<init>(FileInputStream.java:138)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1251)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at sun.misc.Resource.cachedInputStream(Resource.java:77)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at sun.misc.Resource.getByteBuffer(Resource.java:160)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.net.URLClassLoader.access$100(URLClassLoader.java:73)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.net.URLClassLoader$1.run(URLClassLoader.java:368)</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>... 7 more</div>
</div>
<div>
--</div>
<div>
<br /></div>
<div>
<b>Java execution result (testing from console):</b></div>
<div>
<div>
{</div>
<div>
"errorMessage": "Class not found: example.Hello",</div>
<div>
"errorType": "class java.lang.ClassNotFoundException"</div>
<div>
}</div>
</div>
<div>
<br /></div>
<div>
<b>Python CloudWatch logs:</b></div>
<div>
--</div>
<div>
<div>
Unable to import module 'python-hi': No module named python-hi</div>
</div>
<div>
--</div>
<div>
<br /></div>
<div>
<b>Python execution result (testing from console):</b></div>
<div>
<div>
{</div>
<div>
"errorMessage": "Unable to import module 'python-hi'"</div>
<div>
}</div>
</div>
<div>
<br /></div>
<div>
<b>Node CloudWatch logs:</b></div>
<div>
--</div>
<div>
<div>
module initialization error: Error</div>
<div>
at Error (native)</div>
<div>
at Object.fs.openSync (fs.js:549:18)</div>
<div>
at Object.fs.readFileSync (fs.js:397:15)</div>
<div>
at Object.Module._extensions..js (module.js:415:20)</div>
<div>
at Module.load (module.js:343:32)</div>
<div>
at Function.Module._load (module.js:300:12)</div>
<div>
at Module.require (module.js:353:17)</div>
<div>
at require (internal/module.js:12:17)</div>
</div>
<div>
--</div>
<div>
<br /></div>
<div>
<b>Node execution result (testing from console):</b></div>
<div>
<div>
{</div>
<div>
"errorMessage": "EACCES: permission denied, open '/var/task/node-hi.js'",</div>
<div>
"errorType": "Error",</div>
<div>
"stackTrace": [</div>
<div>
"Object.fs.openSync (fs.js:549:18)",</div>
<div>
"Object.fs.readFileSync (fs.js:397:15)",</div>
<div>
"Object.Module._extensions..js (module.js:415:20)",</div>
<div>
"Module.load (module.js:343:32)",</div>
<div>
"Function.Module._load (module.js:300:12)",</div>
<div>
"Module.require (module.js:353:17)",</div>
<div>
"require (internal/module.js:12:17)"</div>
<div>
]</div>
<div>
}</div>
</div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-20249036052493900482016-08-12T18:26:00.000+01:002016-08-12T18:26:02.156+01:00AWS Tip of the day: Tagging EC2 reserved instancesA quick post pointing out that EC2 reserved instances actually support tagging. This functionality is only available on the command line of via the API and not via the console but it still allows to you tag your reservations making it easier to keep track of why a reserved instance was purchased and what component it was intended for. Of course the reservation itself is not actually tied to a running instance in any way, it is merely a billing construct that is applied to any matching instances running in your account but if you are making architectural changes or considering different instance types for specific workloads or components the tags allow you (and your team) to see why the reservation was originally purchased. So for example if you are scaling up the instance sizes of a specific component, lets say from m4.large to m4.xlarge, you can check your reserved instance tags and modify the reservations associated with the component to ensure you continue to benefit from the purchase.<br />
<br />
The tagging of reserved instances works the same as tagging other <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html">EC2 resources</a> through the AWS CLI's ec2 <a href="http://docs.aws.amazon.com/cli/latest/reference/ec2/create-tags.html">create-tags</a> command and specifying the reserved instances ID as the resource ID. You can find the reserved instance ID using the CLI's ec2 <a href="http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-reserved-instances.html">describe-reserved-instances</a> command. Using an actual example, lets start off finding a reservation:<br />
<br />
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">$ aws ec2 describe-reserved-instances</span><br />
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
{</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"ReservedInstances": [</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
{</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"ReservedInstancesId": "3d092b71-5243-4e5e-b409-<wbr></wbr>86df342282ab", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"OfferingType": "No Upfront", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"AvailabilityZone": "eu-west-1c", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"End": "2017-08-12T04:48:58.000Z", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"ProductDescription": "Linux/UNIX", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"UsagePrice": 0.0, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"RecurringCharges": [</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
{</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Amount": 0.01, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Frequency": "Hourly"</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
}</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
], </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Start": "2016-08-12T04:48:59.763Z", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"State": "active", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"FixedPrice": 0.0, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"CurrencyCode": "USD", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Duration": 31536000, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"InstanceTenancy": "default", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"InstanceType": "t2.micro", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"InstanceCount": 1</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
}</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
]</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
}</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<span style="color: black; font-family: "Times New Roman"; font-size: small;">Next, lets add a tag indicating that this reservation is intended for the "production" stack.:</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
$ aws ec2 create-tags --resources 3d092b71-5243-4e5e-b409-<wbr></wbr>86df342282ab --tags Key=Stack,Value=production</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<span style="color: black; font-family: "Times New Roman"; font-size: small;">Checking the result:</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
$ aws ec2 describe-reserved-instances</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
{</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"ReservedInstances": [</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
{</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"ReservedInstancesId": "3d092b71-5243-4e5e-b409-<wbr></wbr>86df342282ab", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"OfferingType": "No Upfront", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"AvailabilityZone": "eu-west-1c", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"End": "2017-08-12T04:48:58.000Z", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"ProductDescription": "Linux/UNIX", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Tags": [</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
{</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Value": "production", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Key": "Stack"</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
}</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
], </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"UsagePrice": 0.0, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"RecurringCharges": [</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
{</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Amount": 0.01, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Frequency": "Hourly"</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
}</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
], </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Start": "2016-08-12T04:48:59.763Z", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"State": "active", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"FixedPrice": 0.0, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"CurrencyCode": "USD", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"Duration": 31536000, </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"InstanceTenancy": "default", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"InstanceType": "t2.micro", </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
"InstanceCount": 1</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
}</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
]</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
}</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<span style="color: black; font-family: "Times New Roman"; font-size: small;">Great we have a tag but what if we have hundreds of reservations, a long list of reservations is not particularly useful for quickly identifying the reservations related to a component or stack. The CLI's <a href="http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html">query and output</a> functionality can help here:</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<span style="color: black; font-family: "Times New Roman"; font-size: small;"><br /></span></div>
<div style="background-color: white; color: #222222;">
<span style="font-family: arial, sans-serif; font-size: 13.3333px;">$ aws ec2 describe-reserved-instances --query 'ReservedInstances[*].{AZ:</span><wbr style="font-size: 13.3333px;"></wbr><span style="font-family: arial, sans-serif; font-size: 13.3333px;">AvailabilityZone,Type:</span><wbr style="font-size: 13.3333px;"></wbr><span style="font-family: arial, sans-serif; font-size: 13.3333px;">InstanceType,Expiry:End,stack:</span><wbr style="font-size: 13.3333px;"></wbr><span style="font-family: arial, sans-serif; font-size: 13.3333px;">Tags[?Key==`Stack`][?Value==`</span><wbr style="font-size: 13.3333px;"></wbr><span style="font-family: arial, sans-serif; font-size: 13.3333px;">production`]}' --output=table</span><br style="font-size: 13.3333px;" /><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">------------------------------<wbr></wbr>--------------------------<br />| DescribeReservedInstances <wbr></wbr> |<br />+-------------+---------------<wbr></wbr>-------------+-----------+<br />| AZ | Expiry | Type |<br />+-------------+---------------<wbr></wbr>-------------+-----------+<br />| eu-west-1c | 2017-08-12T04:48:58.000Z | t2.micro |<br />+-------------+---------------<wbr></wbr>-------------+-----------+</span><br style="font-size: 13.3333px;" /><span style="color: black;">Not quite the console view but easy enough to see that we have one reservation for the "production" Stack.</span></div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-29226573963929997272016-06-07T17:41:00.001+01:002016-06-07T17:41:27.330+01:00AWS Tip: Save S3 costs with abort multipart lifecycle policy<h3>
Introduction</h3>
<div>
S3 multipart uploads provide a number of <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html">benefits</a> -- better throughput, recovery from network errors -- and a number of tools will automatically use multipart uploads for larger uploads. The AWS CLI cp, mv, and sync commands all make use of multipart uploads and make a <a href="http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html#using-s3-commands-managing-objects">note</a> that "If the process is interrupted by a kill command or system failure, the in-progress multipart upload remains in Amazon S3 and must be cleaned up manually..."</div>
<div>
<br /></div>
<div>
The reason you would want to clean up these failed multipart uploads is because you will be charged for the storage they use while waiting for the upload to be completed (or aborted). This post provides some detail on how to find the incomplete uploads and options for removing them to save storage costs.</div>
<div>
<br /></div>
<h3>
Finding incomplete multipart uploads</h3>
<div>
If you have relatively few buckets or only want to check your biggest buckets (<a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html">CloudWatch S3 metrics</a> are useful for finding these) the AWS CLI s3api <a href="http://docs.aws.amazon.com/cli/latest/reference/s3api/list-multipart-uploads.html">list-multipart-uploads</a> command is a simple check:</div>
<blockquote class="tr_bq">
<span style="font-family: "georgia" , "times new roman" , serif;">aws s3api list-multipart-uploads --bucket [bucket-name]</span></blockquote>
No output indicates that the bucket does not contain any incomplete uploads, see the list-multipart-uploads documentation linked above for an example of the output on a bucket that does contain an incomplete upload. A simple bash script to check all your buckets:<br />
<br />
<script src="https://gist.github.com/watchamcb/c412482f1767cf3c3daf08996271a81c.js"></script>
This script will list your buckets and the first page (out of possibly many more) incomplete multipart upload keys along with the date they were initiated. The region lookup is required to handle bucket names containing dots and buckets in eu-central-1 (SigV4).<br />
<br />
<h3>
Cleaning up</h3>
<div>
Once you have identified the buckets containing incomplete uploads it is worth investigating some of the recent failed uploads to see whether there is an underlying issue that needs to be addressed, particularly if the uploads relate to backups or important log files. The most typical cause is instances being terminated before completing uploads (look at <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/lifecycle-hooks.html">Lifecycle Hooks</a> to fix this if you are using Auto Scaling) but they may also be result of applications not performing cleanup on failures or not handling errors correctly.</div>
<div>
<br /></div>
<div>
A multipart upload can be aborted using the <a href="http://docs.aws.amazon.com/cli/latest/reference/s3api/abort-multipart-upload.html">abort-multipart-upload</a> s3api command in the AWS CLI using the object key and upload ID returned by list-multipart-uploads command. This can be scripted but will take time to complete for buckets containing large numbers of incomplete uploads, fortunately there is an easier way. S3 now supports a <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config">bucket lifecycle policy</a> to automatically delete incomplete uploads after a specified period of time. Enabling the policy in the AWS console is fairly quick and easy, see <a href="https://aws.amazon.com/blogs/aws/s3-lifecycle-management-update-support-for-multipart-uploads-and-delete-markers/">Jeff's blog post</a> for details. A rather messy boto3 example script for enabling the policy on all buckets can be found <a href="https://github.com/watchamcb/aws/blob/master/set-abort-multipart.py">here</a>, it should work with most bucket configurations but it comes with no guarantees and you use it at your own risk.</div>
<div>
<br /></div>
<h3>
Conclusion (why you should do this)</h3>
<div>
If you are using S3 to store large (> 5MB) objects and you are spending more than a few dollars a month on S3 storage then there is a fairly good chance that you are paying unnecessarily for failed/incomplete multipart uploads. It should only take a few minutes to review your buckets and could potentially have significant monthly savings.</div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-70633234664078026302016-05-05T12:11:00.004+01:002016-10-12T09:43:56.152+01:00Enabling longer AWS IDs in a region using an IAM roleAWS is moving towards longer EC2 and EBS IDs and you can enable them for an IAM user or at an account level using the root credentials. You can avoid using the root credentials by using an IAM role instead. This is a quick post to explain the steps needed to use an IAM role on an instance to enable the longer IDs at an account level.<br />
<br />
<b>Update: </b>You can now use the --principal-arn argument to to make this change to the root account, see the AWS support post <a href="https://aws.amazon.com/premiumsupport/knowledge-center/enable-long-id/">here</a>.<br />
<br />
<b>Update 2:</b> Another alternative, using this <a href="https://github.com/awslabs/ec2-migrate-longer-id/blob/master/migratelongerids.py">script</a>.<br />
<br />
<b>1. Create a policy allowing modify and describe of ID format</b> (IAM console -> Policies -> Create Policy -> Create Your Own Policy). The following policy document (originally from here with the Deny changed to Allow) provides the necessary permissions,<br />
<br />
{<br />
"Version": "2012-10-17",<br />
"Statement": [<br />
{<br />
"Effect": "Allow",<br />
"Action": [<br />
"ec2:ModifyIdFormat", "ec2:DescribeIdFormat"<br />
],<br />
"Resource": "*"<br />
}<br />
]<br />
}<br />
<br />
<b>2. Create an EC2 role</b> (IAM console -> Roles -> Create New Role -> [Pick a name] -> Amazon EC2). Select the policy name you created in step 1 and attach it to the role.<br />
<br />
<b>3. Launch an instance with the role</b>. It is probably easiest to use an Amazon Linux AMI as the AWS CLI is installed by default. Instance size and region don't really matter.<br />
<br />
<b>4. SSH to the instance and use the CLI to enable the longer IDs</b> in the region that you would like to test against:<br />
$ aws --region eu-west-1 ec2 modify-id-format --resource instance --use-long-ids<br />
<br />
If you don't specify a region the CLI will prompt you to run 'aws configure' to set a default. To check the status of the ID format you can use describe-id-format:<br />
$ aws --region eu-west-1 ec2 describe-id-format<br />
<br />
{<br />
"Statuses": [<br />
{<br />
"UseLongIds": false,<br />
"Resource": "reservation"<br />
},<br />
{<br />
"UseLongIds": true,<br />
"Resource": "instance"<br />
},<br />
{<br />
"UseLongIds": false,<br />
"Resource": "volume"<br />
},<br />
{<br />
"UseLongIds": false,<br />
"Resource": "snapshot"<br />
}<br />
]<br />
}<br />
<br />
<b>5. Longer IDs are now enabled for the account. </b>Resources created by the root account (such as Auto Scaled instances) will be launched with longer IDs.Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-65453643388095623482016-05-04T13:07:00.000+01:002016-05-04T13:07:34.951+01:00'Connection to s3.amazonaws.com timed out' using VPC S3 EndpointA quick post to point you in the right direction if you are getting connection time out errors using the AWS CLI or Boto with <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html">AWS S3 VPC Endpoints</a> in a private subnet. The two most likely causes of this are:<br />
<br />
1. Your bucket name contains dots.<br />
To work around this you can specify the region of the bucket, for example:<br />
aws --region eu-west-1 s3 ls s3://bucket.with.dots/<br />
<br />
2. You are trying to access a bucket in a different region.<br />
This is not supported by VPC Endpoints, see the restrictions <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html#vpc-endpoints-limitations">here</a>.Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-58516015985625724172016-02-18T09:03:00.000+00:002016-02-18T09:03:52.695+00:00Memcached item memory usage<h4>
Introduction</h4>
I was recently involved with an investigation into a Memcached cluster running on AWS ElastiCache and discovered a few interesting details about Memcached. This post looks at the memory overhead of storing items in Memcached and demonstrates the impact of disabling CAS (check and set) on memory utilisation.<br />
<ol>
</ol>
<h4>
Background - Memcached item overhead</h4>
<div>
Before getting into the details some background might be useful. Starting with the basics, how much memory does Memcached actually need to store an item? Lets spin up a cluster to check, for simplicity I am using a single m3.large ElastiCache node (running memcached 1.4.24) which according to the <a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ParameterGroups.Memcached.html#ParameterGroups.Memcached.NodeSpecific">AWS docs</a> provides around 6.2GB of memory for cache items.</div>
<div>
<br /></div>
<div>
The easiest way to look at our new cache is by telnetting to the cluster (from within the VPC) and running commands directly. To start, what does the slab allocation look like:</div>
<div>
<br /></div>
<div>
> stats slabs</div>
<div>
<div>
< STAT active_slabs 0</div>
<div>
< STAT total_malloced 0</div>
<div>
< END</div>
</div>
<div>
<br /></div>
<div>
As expected, our empty cache has 0 active slabs and no memory allocated. Lets add a value and see how this changes things:</div>
<div>
<br /></div>
<div>
<div>
> set k 0 500 1</div>
<div>
> v</div>
<div>
< STORED</div>
</div>
<div>
<br /></div>
<div>
Above, we have set an item with a key of 'k', no flags, a 500 second expiry time and a 1 byte value of 'v'. Looking at the slab stats we now see:</div>
<div>
<br /></div>
<div>
<div>
> stats slabs</div>
<div>
< STAT 1:chunk_size 96</div>
<div>
< STAT 1:chunks_per_page 10922</div>
<div>
< STAT 1:total_pages 1</div>
<div>
< STAT 1:total_chunks 10922</div>
<div>
< STAT 1:used_chunks 1</div>
<div>
< STAT 1:free_chunks 10921</div>
<div>
< STAT 1:free_chunks_end 0</div>
<div>
< STAT 1:mem_requested 67</div>
<div>
< STAT 1:get_hits 0</div>
<div>
< STAT 1:cmd_set 1</div>
<div>
< STAT 1:delete_hits 0</div>
<div>
< STAT 1:incr_hits 0</div>
<div>
< STAT 1:decr_hits 0</div>
<div>
< STAT 1:cas_hits 0</div>
<div>
< STAT 1:cas_badval 0</div>
<div>
< STAT 1:touch_hits 0</div>
<div>
< STAT active_slabs 1</div>
<div>
< STAT total_malloced 1048512</div>
<div>
< END</div>
</div>
<div>
<br /></div>
<div>
Lots of information here but lets focus on the chunks and memory stats for the moment. We now have 1 active slab with a chunk size of 96 bytes and one of these chunks (used_chunks) has been used to store our item, all expected except the relatively large size of the chunk used. We can see the actual memory requested was 67 bytes which means that 65 bytes of overhead are required to store the 2 byte item. Looking into the memcached code we can see that the item struct defined in <a href="https://github.com/memcached/memcached/blob/1.4.24/memcached.h">memcached.h</a> requires 48 bytes plus another 8 bytes for CAS. The remaining 11 bytes consist of:</div>
<div>
<ul>
<li>The null terminated key (2 bytes, 1 byte of overhead, "k\0" in our example)</li>
<li>The CRLF terminated data (3 bytes, 2 bytes of overhead, "v\r\n" in our example)</li>
<li>The item header which is a formatted string containing the flags, data length and terminated with CRLF (6 bytes, the minimum, in our example "_0_1\r\n" with underscores to make the spaces more visible)</li>
</ul>
<div>
The header is created in the item_make_header function in <a href="https://github.com/memcached/memcached/blob/1.4.24/items.c">items.c</a>, the relevant part being:</div>
</div>
<div>
<br /></div>
<div>
<div>
*nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);</div>
<div>
<br /></div>
</div>
<div>
The header formats the flags and data length as integers in a string so this overhead will increase as the data size and flag values increase. For example, setting the flag to 10 will require an extra byte of to store the extra digit in the header:</div>
<div>
<br /></div>
<div>
<div>
> set k 10 500 1</div>
<div>
> v</div>
<div>
< STORED</div>
<div>
> stats slabs</div>
<div>
< STAT 1:chunk_size 96</div>
<div>
< STAT 1:used_chunks 1</div>
<div>
< STAT 1:free_chunks 10921</div>
<div>
< STAT 1:mem_requested 68</div>
<div>
...</div>
<div>
< STAT active_slabs 1</div>
<div>
< STAT total_malloced 1048512</div>
<div>
< END</div>
</div>
<div>
<br /></div>
<div>
So you can expect at least 65 bytes of overhead per item.</div>
<div>
<br /></div>
<h4>
Disable CAS if you don't need it</h4>
<div>
CAS stands for 'Check And Set' which the Memcached <a href="https://github.com/memcached/memcached/blob/1.4.24/doc/protocol.txt">protocol docs</a> describe as: "store this data but only if no one else has updated since I last fetched it."</div>
<div>
<br /></div>
<div>
For a fair number of Memcached use cases CAS will not be used and as a result can be disabled with no impact. You can check your cluster using the stats command and looking at the cas_* values:</div>
<div>
<br /></div>
<div>
> stats</div>
<div>
...</div>
<div>
<div>
< STAT cas_misses 0</div>
<div>
< STAT cas_hits 0</div>
<div>
< STAT cas_badval 0</div>
</div>
<div>
...</div>
<div>
<br /></div>
<div>
Sounds simple enough, lets test it out. After setting cas_disabled to 1 on our cluster and rebooting for the change to take effect, we set our key again:</div>
<div>
<br /></div>
<div>
<div>
> set k 0 500 1</div>
<div>
> v</div>
<div>
< STORED</div>
</div>
<div>
<div>
> stats slabs</div>
<div>
< STAT 1:chunk_size 96</div>
<div>
< STAT 1:used_chunks 1</div>
<div>
< STAT 1:free_chunks 10921</div>
</div>
<div>
<div>
< STAT 1:mem_requested 59</div>
<div>
...</div>
<div>
< STAT active_slabs 1</div>
<div>
< STAT total_malloced 1048512</div>
<div>
< END</div>
</div>
<div>
<br /></div>
<div>
Not much difference except for the requested memory reducing by 8 bytes. Disabling CAS will NOT reduce the wasted memory on its own but it will allow an item to contain an extra 8 bytes of data where previously it would have needed to use a larger chunk which might be wasting significantly more memory. Lets look at an example.</div>
<div>
<br /></div>
<div>
Assume that you are wanting to cache an item of 32 bytes, a 16 byte key and a 16 byte value. From the background section above we know that this will take 66 bytes of overhead pushing the total item size to 98 bytes, 2 bytes more than the chunk size of the first slab (using Memcached defaults), on a CAS enabled node this shows:</div>
<div>
<br /></div>
<div>
<div>
> set 0123456789ABCDEF 0 500 16</div>
<div>
> 0123456789ABCDEF</div>
<div>
< STORED</div>
<div>
> stats slabs</div>
<div>
< STAT 2:chunk_size 120</div>
<div>
< STAT 2:used_chunks 1</div>
<div>
< STAT 2:free_chunks 8737</div>
<div>
< STAT 2:mem_requested 98</div>
<div>
...</div>
<div>
< STAT active_slabs 1</div>
<div>
< STAT total_malloced 1048560</div>
<div>
< END</div>
</div>
<div>
<br /></div>
<div>
Whereas with CAS disabled the overhead is reduced to 58 bytes, allowing the item to fit into the original slab:</div>
<div>
<br /></div>
<div>
<div>
> set 0123456789ABCDEF 0 500 16</div>
<div>
> 0123456789ABCDEF</div>
<div>
< STORED</div>
<div>
> stats slabs</div>
<div>
> STAT 1:chunk_size 96</div>
<div>
> STAT 1:used_chunks 1</div>
<div>
> STAT 1:free_chunks 10921</div>
<div>
> STAT 1:mem_requested 90</div>
<div>
...</div>
<div>
> STAT active_slabs 1</div>
<div>
> STAT total_malloced 1048512</div>
<div>
> END</div>
</div>
<div>
<br /></div>
<div>
This means that with CAS enabled we are wasting 18% of the memory in each chunk of this slab (22 out of 120 bytes lost per item) compared to just 6% (6 out of 96 bytes) lost with CAS disabled. If the entire cache consisted of items this size, it would result in more than 1 GB of wasted memory compared to 372 MB wasted with CAS disabled. To put this differently, disabling CAS would allow you to store approximately 7 million additional 90 byte items resulting in better cache hit rates and lower eviction rates. This is of course an artificial example so lets look at a more realistic example where the cache item sizes are not quite so uniform.<br />
<br />
The following stats are based on a production CAS enabled node's stats adjusted by a factor of ten to fit on an m3.large (the production node had around 59 million items in slab 11 this was scaled to 5.9 million for testing). The test code used to generate this data is relatively simple and distributes the items in a slab fairly evenly across the item sizes of a slab, each item's size is slightly weighted to try and match the average item size in production.<br />
<br />
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="border-collapse: collapse; border: 1px solid #ccc; font-family: Calibri; font-size: 13px; table-layout: fixed;"><colgroup><col width="31"></col><col width="57"></col><col width="68"></col><col width="96"></col><col width="97"></col><col width="63"></col><col width="96"></col></colgroup><tbody>
<tr style="height: 20px;"><td data-sheets-value="[null,2,"Slab"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Slab</td><td data-sheets-value="[null,2,"Chunk Size"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Chunk Size</td><td data-sheets-value="[null,2,"Chunks Used"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Chunks Used</td><td data-sheets-value="[null,2,"Memory Allocated"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Memory Allocated</td><td data-sheets-value="[null,2,"Memory Requested"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Memory Requested</td><td data-sheets-value="[null,2,"Average Item Size"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Average Item Size</td><td data-sheets-value="[null,2,"Memory Wasted"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Memory Wasted</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,4]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4</td><td data-sheets-value="[null,3,null,192]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">192</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,139810]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">139,810</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,26843520]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">26,843,520</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,25706546]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">25,706,546</td><td data-sheets-value="[null,3,null,184]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">184</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1136974]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,136,974</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,5]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5</td><td data-sheets-value="[null,3,null,240]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">240</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,55924]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">55,924</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,13421760]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13,421,760</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11399776]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,399,776</td><td data-sheets-value="[null,3,null,204]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">204</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2021984]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,021,984</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,10]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">10</td><td data-sheets-value="[null,3,null,752]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">752</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,232025]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">232,025</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,174482800]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">174,482,800</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,162673515]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">162,673,515</td><td data-sheets-value="[null,3,null,701]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">701</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11809285]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,809,285</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,11]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11</td><td data-sheets-value="[null,3,null,944]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">944</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5985735]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5,985,735</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5650533840]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5,650,533,840</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4643546639]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4,643,546,639</td><td data-sheets-value="[null,3,null,776]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">776</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1006987201]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,006,987,201</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,12]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">12</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1184]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,184</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,10951]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">10,951</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,12965984]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">12,965,984</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11429965]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,429,965</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1044]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,044</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1536019]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,536,019</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,13]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1480]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,480</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,7177]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">7,177</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,10621960]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">10,621,960</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,9421613]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">9,421,613</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1313]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,313</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1200347]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,200,347</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,14]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">14</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1856]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,856</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6071]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">6,071</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11267776]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,267,776</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,10148664]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">10,148,664</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1672]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,672</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1119112]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,119,112</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,15]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">15</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2320]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,320</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5325]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5,325</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,12354000]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">12,354,000</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11094291]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,094,291</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2083]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,083</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1259709]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,259,709</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,16]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">16</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2904]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,904</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4621]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4,621</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,13419384]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13,419,384</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11947574]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,947,574</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2585]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,585</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1471810]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,471,810</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,17]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">17</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3632]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,632</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3695]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,695</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,13420240]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13,420,240</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11906634]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,906,634</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3222]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,222</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1513606]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,513,606</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,18]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">18</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4544]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4,544</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5201]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5,201</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23633344]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23,633,344</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,21025609]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">21,025,609</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4043]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4,043</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2607735]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,607,735</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,19]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">19</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5680]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5,680</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4523]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4,523</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,25690640]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">25,690,640</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23038012]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23,038,012</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5094]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5,094</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2652628]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,652,628</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,20]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">20</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,7104]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">7,104</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3778]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,778</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,26838912]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">26,838,912</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23778017]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23,778,017</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6294]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">6,294</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3060895]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,060,895</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,21]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">21</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,8880]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">8,880</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3022]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,022</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,26835360]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">26,835,360</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23630979]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23,630,979</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,7820]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">7,820</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3204381]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,204,381</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,22]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">22</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11104]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,104</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2417]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,417</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,26838368]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">26,838,368</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23754944]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23,754,944</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,9828]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">9,828</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3083424]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,083,424</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,23]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,13880]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13,880</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1933]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,933</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,26830040]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">26,830,040</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23092970]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23,092,970</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11947]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11,947</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3737070]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,737,070</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,24]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">24</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,17352]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">17,352</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1107]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,107</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,19208664]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">19,208,664</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,15554075]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">15,554,075</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,14051]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">14,051</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3654589]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3,654,589</td></tr>
<tr style="height: 20px;"><td style="padding: 0px 3px 0px 3px; vertical-align: bottom;"></td><td style="padding: 0px 3px 0px 3px; vertical-align: bottom;"></td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6473315]" style="font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">6,473,315</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6115206592]" style="font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">6,115,206,592</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5063149823]" style="font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5,063,149,823</td><td style="padding: 0px 3px 0px 3px; vertical-align: bottom;"></td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1052056769]" style="font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,052,056,769</td></tr>
</tbody></table>
</div>
<div>
<br />
The table shows that we are caching around 6.4 million items with the majority of them falling in slabs 4, 10, and 11 and that around 1 GB of memory is being wasted. Average item size is computed from (memory requested / chunks used, rounded to the nearest byte) and is an indicator of the actual item sizes (including overhead) within each slab. An interesting point relevant to the CAS discussion is that the average item size for slab 11 is 776 bytes which is only marginally bigger than the chunk size for slab 10.<br />
<br />
Running the test code against a node with CAS disabled yields the following results:<br />
<br />
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="border-collapse: collapse; border: 1px solid rgb(204, 204, 204); font-family: Calibri; font-size: 13px; table-layout: fixed;"><colgroup><col width="31"></col><col width="54"></col><col width="71"></col><col width="93"></col><col width="92"></col><col width="64"></col><col width="82"></col></colgroup><tbody>
<tr style="height: 20px;"><td data-sheets-value="[null,2,"Slab"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Slab</td><td data-sheets-value="[null,2,"Chunk Size"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Chunk Size</td><td data-sheets-value="[null,2,"Chunks Used"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Chunks Used</td><td data-sheets-value="[null,2,"Memory Allocated"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Memory Allocated</td><td data-sheets-value="[null,2,"Memory Requested"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Memory Requested</td><td data-sheets-value="[null,2,"Average Item Size"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Average Item Size</td><td data-sheets-value="[null,2,"Memory Wasted"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom; word-wrap: break-word;">Memory Wasted</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,4]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">4</td><td data-sheets-value="[null,3,null,192]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">192</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,165990]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">165,990</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,31870080]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">31,870,080</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,29471729]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">29,471,729</td><td data-sheets-value="[null,3,null,178]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">178</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2398351]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,398,351</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,5]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">5</td><td data-sheets-value="[null,3,null,240]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">240</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,29744]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">29,744</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,7138560]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">7,138,560</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6068721]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">6,068,721</td><td data-sheets-value="[null,3,null,204]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">204</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1069839]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,069,839</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,10]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">10</td><td data-sheets-value="[null,3,null,752]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">752</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3365926]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">3,365,926</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2531176352]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,531,176,352</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2496542132]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,496,542,132</td><td data-sheets-value="[null,3,null,742]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">742</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,34634220]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">34,634,220</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,11]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">11</td><td data-sheets-value="[null,3,null,944]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">944</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2853079]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,853,079</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2693306576]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,693,306,576</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2261103795]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,261,103,795</td><td data-sheets-value="[null,3,null,793]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">793</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,432202781]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">432,202,781</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,12]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">12</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1184]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,184</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,10331]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">10,331</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,12231904]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">12,231,904</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,10910829]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">10,910,829</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1056]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,056</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1321075]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,321,075</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,13]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">13</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1480]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,480</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6603]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">6,603</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,9772440]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">9,772,440</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,8703301]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">8,703,301</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1318]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,318</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1069139]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,069,139</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,14]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">14</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1856]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,856</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6116]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">6,116</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11351296]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">11,351,296</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,10202507]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">10,202,507</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1668]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,668</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1148789]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,148,789</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,15]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">15</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2320]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,320</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5469]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">5,469</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,12688080]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">12,688,080</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11429195]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">11,429,195</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2090]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,090</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1258885]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,258,885</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,16]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">16</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2904]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,904</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4651]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">4,651</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,13506504]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">13,506,504</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,12137620]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">12,137,620</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2610]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,610</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1368884]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,368,884</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,17]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">17</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3632]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">3,632</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3629]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">3,629</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,13180528]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">13,180,528</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11834384]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">11,834,384</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3261]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">3,261</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1346144]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,346,144</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,18]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">18</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4544]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">4,544</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5089]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">5,089</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23124416]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">23,124,416</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,20661849]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">20,661,849</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4060]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">4,060</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2462567]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,462,567</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,19]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">19</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5680]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">5,680</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,4557]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">4,557</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,25883760]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">25,883,760</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23299194]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">23,299,194</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5113]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">5,113</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2584566]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,584,566</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,20]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">20</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,7104]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">7,104</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3786]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">3,786</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,26895744]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">26,895,744</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23983965]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">23,983,965</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6335]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">6,335</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2911779]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,911,779</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,21]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">21</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,8880]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">8,880</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3056]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">3,056</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,27137280]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">27,137,280</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,24146469]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">24,146,469</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,7901]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">7,901</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2990811]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,990,811</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,22]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">22</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,11104]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">11,104</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2376]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,376</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,26383104]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">26,383,104</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,23654235]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">23,654,235</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,9955]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">9,955</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2728869]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,728,869</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,23]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">23</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,13880]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">13,880</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,2297]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">2,297</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,31882360]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">31,882,360</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,28479830]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">28,479,830</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,12399]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">12,399</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,3402530]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">3,402,530</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,24]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">24</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,17352]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">17,352</td><td data-sheets-value="[null,3,null,616]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">616</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,10688832]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">10,688,832</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,8733548]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">8,733,548</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,14178]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">14,178</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,1955284]" style="padding: 0px 3px; text-align: right; vertical-align: bottom;">1,955,284</td></tr>
<tr style="height: 20px;"><td style="padding: 0px 3px; vertical-align: bottom;"></td><td style="padding: 0px 3px; vertical-align: bottom;"></td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,6473315]" style="font-weight: bold; padding: 0px 3px; text-align: right; vertical-align: bottom;">6,473,315</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5508217816]" style="font-weight: bold; padding: 0px 3px; text-align: right; vertical-align: bottom;">5,508,217,816</td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,5011363303]" style="font-weight: bold; padding: 0px 3px; text-align: right; vertical-align: bottom;">5,011,363,303</td><td style="padding: 0px 3px; vertical-align: bottom;"></td><td data-sheets-numberformat="[null,2,"#,##0",1]" data-sheets-value="[null,3,null,496854513]" style="font-weight: bold; padding: 0px 3px; text-align: right; vertical-align: bottom;">496,854,513</td></tr>
</tbody></table>
<br />
As expected, the extra 8 bytes per slab has allowed a large number of items to be moved to slabs having smaller chunk sizes with more than half of the items in slab 11 now fitting into slab 10 and saving almost 600 MB of memory. The average item size being closer to the slab chunk size is another indicator that we are making more efficient use of the memory in each slab.<br />
<br />
For reference, the size of the items each slab can contain are below. The chunk sizes are displayed when running 'memcached -vv', reformatting this into a table and calculating the maximum item size gives:<br />
<br />
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="border-collapse: collapse; border: 1px solid #ccc; font-family: Calibri; font-size: 13px; table-layout: fixed;"><colgroup><col width="31"></col><col width="69"></col><col width="62"></col><col width="61"></col></colgroup><tbody>
<tr style="height: 20px;"><td data-sheets-value="[null,2,"Slab"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom;">Slab</td><td data-sheets-value="[null,2,"Chunk size"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom;">Chunk size</td><td data-sheets-value="[null,2,"Max data"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom;">Max data</td><td data-sheets-value="[null,2,"Min Data"]" style="font-weight: bold; padding: 0px 3px; vertical-align: bottom;">Min data</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,1]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1</td><td data-sheets-value="[null,3,null,96]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">96</td><td data-sheets-value="[null,3,null,30]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">30</td><td data-sheets-value="[null,3,null,1]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,2]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2</td><td data-sheets-value="[null,3,null,120]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">120</td><td data-sheets-value="[null,3,null,54]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">54</td><td data-sheets-value="[null,3,null,31]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">31</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,3]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3</td><td data-sheets-value="[null,3,null,152]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">152</td><td data-sheets-value="[null,3,null,86]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">86</td><td data-sheets-value="[null,3,null,55]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">55</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,4]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4</td><td data-sheets-value="[null,3,null,192]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">192</td><td data-sheets-value="[null,3,null,125]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">125</td><td data-sheets-value="[null,3,null,87]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">87</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,5]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5</td><td data-sheets-value="[null,3,null,240]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">240</td><td data-sheets-value="[null,3,null,173]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">173</td><td data-sheets-value="[null,3,null,126]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">126</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,6]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">6</td><td data-sheets-value="[null,3,null,304]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">304</td><td data-sheets-value="[null,3,null,237]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">237</td><td data-sheets-value="[null,3,null,174]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">174</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,7]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">7</td><td data-sheets-value="[null,3,null,384]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">384</td><td data-sheets-value="[null,3,null,317]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">317</td><td data-sheets-value="[null,3,null,238]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">238</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,8]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">8</td><td data-sheets-value="[null,3,null,480]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">480</td><td data-sheets-value="[null,3,null,413]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">413</td><td data-sheets-value="[null,3,null,318]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">318</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,9]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">9</td><td data-sheets-value="[null,3,null,600]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">600</td><td data-sheets-value="[null,3,null,533]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">533</td><td data-sheets-value="[null,3,null,414]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">414</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,10]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">10</td><td data-sheets-value="[null,3,null,752]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">752</td><td data-sheets-value="[null,3,null,685]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">685</td><td data-sheets-value="[null,3,null,534]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">534</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,11]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11</td><td data-sheets-value="[null,3,null,944]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">944</td><td data-sheets-value="[null,3,null,877]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">877</td><td data-sheets-value="[null,3,null,686]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">686</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,12]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">12</td><td data-sheets-value="[null,3,null,1184]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1184</td><td data-sheets-value="[null,3,null,1116]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1116</td><td data-sheets-value="[null,3,null,878]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">878</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,13]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13</td><td data-sheets-value="[null,3,null,1480]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1480</td><td data-sheets-value="[null,3,null,1412]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1412</td><td data-sheets-value="[null,3,null,1117]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1117</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,14]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">14</td><td data-sheets-value="[null,3,null,1856]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1856</td><td data-sheets-value="[null,3,null,1788]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1788</td><td data-sheets-value="[null,3,null,1413]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1413</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,15]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">15</td><td data-sheets-value="[null,3,null,2320]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2320</td><td data-sheets-value="[null,3,null,2252]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2252</td><td data-sheets-value="[null,3,null,1789]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1789</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,16]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">16</td><td data-sheets-value="[null,3,null,2904]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2904</td><td data-sheets-value="[null,3,null,2836]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2836</td><td data-sheets-value="[null,3,null,2253]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2253</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,17]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">17</td><td data-sheets-value="[null,3,null,3632]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3632</td><td data-sheets-value="[null,3,null,3564]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3564</td><td data-sheets-value="[null,3,null,2837]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2837</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,18]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">18</td><td data-sheets-value="[null,3,null,4544]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4544</td><td data-sheets-value="[null,3,null,4476]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4476</td><td data-sheets-value="[null,3,null,3565]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3565</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,19]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">19</td><td data-sheets-value="[null,3,null,5680]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5680</td><td data-sheets-value="[null,3,null,5612]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5612</td><td data-sheets-value="[null,3,null,4477]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">4477</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,20]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">20</td><td data-sheets-value="[null,3,null,7104]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">7104</td><td data-sheets-value="[null,3,null,7036]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">7036</td><td data-sheets-value="[null,3,null,5613]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5613</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,21]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">21</td><td data-sheets-value="[null,3,null,8880]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">8880</td><td data-sheets-value="[null,3,null,8812]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">8812</td><td data-sheets-value="[null,3,null,7037]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">7037</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,22]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">22</td><td data-sheets-value="[null,3,null,11104]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11104</td><td data-sheets-value="[null,3,null,11035]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11035</td><td data-sheets-value="[null,3,null,8813]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">8813</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,23]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">23</td><td data-sheets-value="[null,3,null,13880]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13880</td><td data-sheets-value="[null,3,null,13811]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13811</td><td data-sheets-value="[null,3,null,11036]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">11036</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,24]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">24</td><td data-sheets-value="[null,3,null,17352]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">17352</td><td data-sheets-value="[null,3,null,17283]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">17283</td><td data-sheets-value="[null,3,null,13812]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">13812</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,25]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">25</td><td data-sheets-value="[null,3,null,21696]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">21696</td><td data-sheets-value="[null,3,null,21627]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">21627</td><td data-sheets-value="[null,3,null,17284]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">17284</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,26]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">26</td><td data-sheets-value="[null,3,null,27120]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">27120</td><td data-sheets-value="[null,3,null,27051]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">27051</td><td data-sheets-value="[null,3,null,21628]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">21628</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,27]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">27</td><td data-sheets-value="[null,3,null,33904]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">33904</td><td data-sheets-value="[null,3,null,33835]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">33835</td><td data-sheets-value="[null,3,null,27052]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">27052</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,28]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">28</td><td data-sheets-value="[null,3,null,42384]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">42384</td><td data-sheets-value="[null,3,null,42315]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">42315</td><td data-sheets-value="[null,3,null,33836]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">33836</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,29]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">29</td><td data-sheets-value="[null,3,null,52984]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">52984</td><td data-sheets-value="[null,3,null,52915]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">52915</td><td data-sheets-value="[null,3,null,42316]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">42316</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,30]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">30</td><td data-sheets-value="[null,3,null,66232]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">66232</td><td data-sheets-value="[null,3,null,66163]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">66163</td><td data-sheets-value="[null,3,null,52916]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">52916</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,31]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">31</td><td data-sheets-value="[null,3,null,82792]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">82792</td><td data-sheets-value="[null,3,null,82723]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">82723</td><td data-sheets-value="[null,3,null,66164]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">66164</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,32]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">32</td><td data-sheets-value="[null,3,null,103496]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">103496</td><td data-sheets-value="[null,3,null,103426]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">103426</td><td data-sheets-value="[null,3,null,82724]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">82724</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,33]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">33</td><td data-sheets-value="[null,3,null,129376]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">129376</td><td data-sheets-value="[null,3,null,129306]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">129306</td><td data-sheets-value="[null,3,null,103427]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">103427</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,34]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">34</td><td data-sheets-value="[null,3,null,161720]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">161720</td><td data-sheets-value="[null,3,null,161650]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">161650</td><td data-sheets-value="[null,3,null,129307]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">129307</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,35]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">35</td><td data-sheets-value="[null,3,null,202152]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">202152</td><td data-sheets-value="[null,3,null,202082]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">202082</td><td data-sheets-value="[null,3,null,161651]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">161651</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,36]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">36</td><td data-sheets-value="[null,3,null,252696]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">252696</td><td data-sheets-value="[null,3,null,252626]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">252626</td><td data-sheets-value="[null,3,null,202083]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">202083</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,37]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">37</td><td data-sheets-value="[null,3,null,315872]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">315872</td><td data-sheets-value="[null,3,null,315802]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">315802</td><td data-sheets-value="[null,3,null,252627]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">252627</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,38]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">38</td><td data-sheets-value="[null,3,null,394840]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">394840</td><td data-sheets-value="[null,3,null,394770]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">394770</td><td data-sheets-value="[null,3,null,315803]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">315803</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,39]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">39</td><td data-sheets-value="[null,3,null,493552]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">493552</td><td data-sheets-value="[null,3,null,493482]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">493482</td><td data-sheets-value="[null,3,null,394771]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">394771</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,40]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">40</td><td data-sheets-value="[null,3,null,616944]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">616944</td><td data-sheets-value="[null,3,null,616874]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">616874</td><td data-sheets-value="[null,3,null,493483]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">493483</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,41]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">41</td><td data-sheets-value="[null,3,null,771184]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">771184</td><td data-sheets-value="[null,3,null,771114]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">771114</td><td data-sheets-value="[null,3,null,616875]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">616875</td></tr>
<tr style="height: 20px;"><td data-sheets-value="[null,3,null,42]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">42</td><td data-sheets-value="[null,3,null,1048576]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1048576</td><td data-sheets-value="[null,3,null,1048505]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1048505</td><td data-sheets-value="[null,3,null,771115]" style="padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">771115</td></tr>
</tbody></table>
<br />
This table is using the default Memcached config settings (48 min item, 1.25 growth factor, cas enabled). The "Max data" column is the maximum number of bytes that can be used to store both the key and the value for an item in this slab. The "Min data" column is the minimum data size required for an item to be put in a slab and is simply the maximum data size from the previous slab + 1. With CAS disabled the max and min data values would be increased by 8 bytes for all items with the exception of the minimum data for slab 1 remaining at 1 byte. The following Python function can be used for calculating the maximum item size (key + value) per slab:<br />
<br />
<br /></div>
<script src="https://gist.github.com/watchamcb/8625618560c94eafb3e7.js"></script>
<br />
<h4>
Conclusion</h4>
<div>
In this post we have had a look at the details of Memcached item overhead and identified disabling CAS as a simple mechanism of improving memory utilisation which will reduce evictions and increase hit rate. In the next post we will look at how adjusting the growth factor and minimum item size can further improve memory utilisation.</div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-83759303438496030532015-07-13T15:58:00.000+01:002015-07-13T16:34:44.162+01:00C-states and P-states with Ubuntu 14.04 on Amazon EC2<h3>
Introduction</h3>
The largest new generation EC2 instance types now expose <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processor_state_control.html">C-state and P-state</a> control to the operating system, unfortunately additional complexity is introduced with the added control and this post will discuss some of the issues related to this on Ubuntu 14.04.<br />
<br />
For comparison I am going to be using a relatively vanilla Ubuntu 14.04.02 (3.13.0-57-generic) AMI running memcached on both a c3.8xl and a c4.8xl. As a memcached workload is generally dependant on memory and network, I have enabled Receive Packet Steering (RPS) as described by <a href="http://engineering.pinterest.com/post/53467339970/building-pinterest-in-the-cloud">this</a> Pinterest article and ensuring it persists across reboots by adding it in /etc/rc.local.<br />
<br />
<h3>
More haste, less speed (TL;DR)</h3>
<div>
For those without the time or patience to read the full post, intel_pstate is disabled by default in Ubuntu 14.04 and the ondemand rc.d script will prevent optimal performance of a latency sensitive application on C-state/P-state enabled instances. Make these changes to improve latency and reduce variability:</div>
<div>
<br /></div>
<div>
1. Disable ondemand (dynamic) CPU frequency scaling</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">$ sudo update-rc.d ondemand disable</span></div>
<div>
<div>
<div>
<br /></div>
<div>
2. Edit /etc/default/grub.d/50-cloudimg-settings.cfg and update the Grub command line to be:</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;"># Set the default commandline</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 intel_idle.max_cstate=1 intel_pstate=enable"</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
3. Install the new version of grub</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">$ sudo update-grub</span></div>
</div>
<div>
<br /></div>
<div>
4. Reboot</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo reboot</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<h3>
Not so idle (for those with time)</h3>
The first notable thing is that the idle state on the c4.8xl differs from the c3.8xl according to CloudWatch:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUcb0oZb0rXglsBSfx9jsmuAUrHelnRb0jyzD07kNitMQgjSvFgBNSQ-gwvwJXtwF6WeOCBwXolcAEjuS70pZgO7Jq-8w0SCo4C8Ca66UtbP7qxqrGIb0MokxHkiSFM88LPDGKf6g_3ec/s1600/idle.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUcb0oZb0rXglsBSfx9jsmuAUrHelnRb0jyzD07kNitMQgjSvFgBNSQ-gwvwJXtwF6WeOCBwXolcAEjuS70pZgO7Jq-8w0SCo4C8Ca66UtbP7qxqrGIb0MokxHkiSFM88LPDGKf6g_3ec/s640/idle.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Idle instances c3.8xl (i-2cb4bd86) vs. c4.8xl (i-0e737ba4)</td></tr>
</tbody></table>
<br />
According to sar the c3 is marginally more idle:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sar 1 10</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Linux 3.13.0-57-generic (ip-172-31-40-39) <span class="Apple-tab-span" style="white-space: pre;"> </span>07/12/2015 <span class="Apple-tab-span" style="white-space: pre;"> </span>_x86_64_<span class="Apple-tab-span" style="white-space: pre;"> </span>(32 CPU)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:51 PM CPU %user %nice %system %iowait %steal %idle</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:52 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:53 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:54 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:55 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:56 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:57 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:58 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:59 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:12:00 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:12:01 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<div>
<br /></div>
<br />
While the c4 is doing some system related work:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sar 1 10</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Linux 3.13.0-57-generic (ip-172-31-34-50) <span class="Apple-tab-span" style="white-space: pre;"> </span>07/12/2015 <span class="Apple-tab-span" style="white-space: pre;"> </span>_x86_64_<span class="Apple-tab-span" style="white-space: pre;"> </span>(36 CPU)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:46 PM CPU %user %nice %system %iowait %steal %idle</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:47 PM all 0.00 0.00 0.03 0.00 0.00 99.97</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:48 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:49 PM all 0.00 0.00 0.03 0.00 0.00 99.97</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:50 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:51 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:52 PM all 0.03 0.00 0.03 0.00 0.00 99.94</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:53 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:54 PM all 0.00 0.00 0.03 0.00 0.00 99.97</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:55 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">05:11:56 PM all 0.00 0.00 0.00 0.00 0.00 100.00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 0.00 0.00 0.01 0.00 0.00 99.99</span><br />
<div>
<br /></div>
Although this does not account for the ~0.5% CPU being reported by CloudWatch, presumably due to some overhead for C-state management. On the c3 we don't seem to have a cpuidle driver:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /sys/devices/system/cpu/cpuidle/current_driver</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">none</span><br />
<br />
Whereas on the c4 we have:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /sys/devices/system/cpu/cpuidle/current_driver</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">intel_idle</span><br />
<div>
<br /></div>
<h3>
Disabling C-states (intel_idle driver)</h3>
<div>
Out of interest lets see what the impact of disabling the driver is. In order to do this we need to set some kernel parameters in Grub which live in /etc/default/grub.d/50-cloudimg-settings.cfg on Ubuntu instances running on EC2, the updated file should look like this:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /etc/default/grub.d/50-cloudimg-settings.cfg</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Cloud Image specific Grub settings for Generic Cloud Images</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># CLOUD_IMG: This file was created/modified by the Cloud Image build process</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Set the recordfail timeout</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_RECORDFAIL_TIMEOUT=0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Do not wait on grub prompt</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_TIMEOUT=0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Set the default commandline</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 intel_idle.max_cstate=0 processor.max_cstate=0"</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Set the grub console type</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_TERMINAL=console</span></div>
</div>
<div>
<br /></div>
<div>
And can be installed with:</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo update-grub</span></div>
<div>
<br /></div>
<div>
A relatively good explanation of the <a href="https://www.kernel.org/doc/Documentation/kernel-parameters.txt">kernel options</a> can be found on <a href="http://stackoverflow.com/questions/12111954/context-switches-much-slower-in-new-linux-kernels">stackoverflow</a>, note that the idle=poll option will cause CloudWatch to report 100% CPU and is not advised. After a reboot for the changes to take effect the following CloudWatch graph (including a default c4.8xl for comparison) shows the difference in idle CPU.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjimmWhtbKwD1_XTqC6mGFcSGXgObrSXspb6L5hsyCnkkx4zK9J0GFBsSiRmY6YXB55B5lelDE1HsJltfKcgcc1i72mLA8-kDHw-pal5trPWZvIB5qBPMTXhUoBQrIGQoAV93Daw31qcAM/s1600/idle-no-cstate.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjimmWhtbKwD1_XTqC6mGFcSGXgObrSXspb6L5hsyCnkkx4zK9J0GFBsSiRmY6YXB55B5lelDE1HsJltfKcgcc1i72mLA8-kDHw-pal5trPWZvIB5qBPMTXhUoBQrIGQoAV93Daw31qcAM/s640/idle-no-cstate.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: 12.8000001907349px;">Idle instances c3.8xl (i-2cb4bd86) vs. c4.8xl with C-state disabled (i-0e737ba4)</span></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
A notable improvement but still not identical, lets take a look at performance next.</div>
<div>
<br /></div>
<h3>
The need for speed</h3>
<div>
As the c3.8xl does not support P-states we would not expect to see a frequency scaling policy or any difference between the core CPU frequencies and this can be confirmed with the cpupower utility:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo cpupower frequency-info</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">analyzing CPU 0:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> no or unknown cpufreq driver is active on this CPU</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> boost state support:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Supported: no</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Active: no</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 25500 MHz max turbo 4 active cores</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 25500 MHz max turbo 3 active cores</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 25500 MHz max turbo 2 active cores</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 25500 MHz max turbo 1 active cores</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"></span><br />
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">
</span>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /proc/cpuinfo | grep MHz</span></div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">
<div>
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
[...]<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044<br />
cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2800.044</div>
<div>
<br /></div>
</span></div>
<div>
<br /></div>
<div>
Nothing terribly surprising here, no cpufreq driver and all the cores running at the default frequency. Looking at the c4.8xl however, we see a few noteworthy items:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo cpupower frequency-info</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">analyzing CPU 0:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> driver: acpi-cpufreq</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> CPUs which run at the same hardware frequency: 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> CPUs which need to have their frequency coordinated by software: 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> maximum transition latency: 10.0 us.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> hardware limits: 1.20 GHz - 2.90 GHz</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> available frequency steps: 2.90 GHz, 2.90 GHz, 2.80 GHz, 2.70 GHz, 2.50 GHz, 2.40 GHz, 2.30 GHz, 2.20 GHz, 2.00 GHz, 1.90 GHz, 1.80 GHz, 1.70 GHz, 1.60 GHz, 1.40 GHz, 1.30 GHz, 1.20 GHz</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> available cpufreq governors: conservative, ondemand, userspace, powersave, performance</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> current policy: frequency should be within 1.20 GHz and 2.90 GHz.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> The governor "ondemand" may decide which speed to use</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> within this range.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> current CPU frequency is 1.20 GHz (asserted by call to hardware).</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> cpufreq stats: 2.90 GHz:1.17%, 2.90 GHz:0.00%, 2.80 GHz:0.00%, 2.70 GHz:0.01%, 2.50 GHz:0.00%, 2.40 GHz:0.00%, 2.30 GHz:0.00%, 2.20 GHz:0.00%, 2.00 GHz:0.00%, 1.90 GHz:0.00%, 1.80 GHz:0.00%, 1.70 GHz:0.00%, 1.60 GHz:0.00%, 1.40 GHz:0.00%, 1.30 GHz:0.00%, 1.20 GHz:98.82% (15)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> boost state support:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Supported: yes</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Active: yes</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /proc/cpuinfo | grep MHz</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">[...]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1600.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 1200.000</span></div>
</div>
<div>
<br /></div>
<div>
Firstly, it seems Ubuntu is using the acpi-cpufreq driver instead of the new and improved intel_pstate driver (more on this later). Secondly we are using the 'ondemand' governor which is great for power saving but not ideal for a high performance or low latency server. Finally, thanks to the ondemand governor all of the cores are running significantly slower than their advertised (3.2 GHz) maximum. Some digging finds that it is related to <a href="https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/1314653">this</a> and is relatively easy to fix by disabling /etc/init.d/ondemand:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo update-rc.d ondemand disable</span></div>
<div>
<br /></div>
<div>
Which will prevent the 'performance' governor from being replaced with 'ondemand' at the next boot. To correct the frequency without a reboot we can update the governor with cpupower:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo cpupower frequency-set -g performance</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /proc/cpuinfo | grep MHz</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">[...]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 2901.000</span></div>
</div>
<div>
<br /></div>
<div>
Still not hitting the maximum frequency but this shaves a few more fractions off the idle CPU usage reported by CloudWatch:</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhopr0pTfiUQPecGA0yCdllVx2c6lIXiXxvdX-FLklLnYNh8FiHzL6AD0AMTpDrlSF7CuDqFuCCb257HDe9LG3U6-7TiCVg_8uj3hxJrZnISEOT8QO2A6KrguhGqiuYrKWUXcps3XqZY7k/s1600/idle-fix.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="396" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhopr0pTfiUQPecGA0yCdllVx2c6lIXiXxvdX-FLklLnYNh8FiHzL6AD0AMTpDrlSF7CuDqFuCCb257HDe9LG3U6-7TiCVg_8uj3hxJrZnISEOT8QO2A6KrguhGqiuYrKWUXcps3XqZY7k/s640/idle-fix.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: 12.8000001907349px;">Idle instances c3.8xl (i-2cb4bd86) vs. c4.8xl with C-state disabled and performance frequency policy (i-0e737ba4)</span></td></tr>
</tbody></table>
<br />
<h3>
Adding some stress</h3>
<div>
Thus far we have been looking at the idle CPU utilisation and have got relatively close to the same baseline performance with our customised c4.8xl. Time now to look at how adding some load changes things, starting of with a relatively simple test using 'stress' to generate artificial load on CPU and memory.</div>
<div>
<br /></div>
<div>
Using 'stress -c 4 -m 2' the c3.8xl shows the following sar average after running for a few minutes:</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">06:58:11 AM CPU %user %nice %system %iowait %steal %idle</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">06:58:40 AM all 12.65 0.00 6.12 0.00 0.00 81.23</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 12.63 0.06 6.14 0.03 0.00 81.14</span></div>
</div>
<div>
<br /></div>
<div>
The default c4.8xl shows:</div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">06:58:11 AM CPU %user %nice %system %iowait %steal %idle</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">06:58:41 AM all 11.22 0.00 5.50 0.00 0.00 83.28</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 11.24 0.15 5.47 0.03 0.00 83.10</span></div>
</div>
<div>
<br /></div>
<div>
And the tweaked c4.8xl (with C-states disabled and performance policy) shows:</div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">06:58:11 AM CPU %user %nice %system %iowait %steal %idle</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">06:58:43 AM all 11.27 0.00 5.44 0.00 0.00 83.29</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 11.23 0.00 5.45 0.00 0.00 83.32</span></div>
</div>
<div>
<br /></div>
<div>
The CloudWatch graph shows the c3 (i-2cb4bd86) averaging around 18.8%, the default c4 (i-c2b4bd68) averaging around 17.1%, and the tweaked c4 (i-0e737ba4) averaging around 16.8%. All of which match the sar %idle relatively closely:</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimqhI9GnXt8LS1x4-K4aAzWIFvtirLBeOMXbvfZdQvl8rgIu9Lvo7S0XhEj7Czg_hAmGpsPkHM4iDyaqEJRuZjX0ViHtNejHeveujL11Fe0rHDD9BRcGabROd-Hm-TNaOm1mqv4i_9D-g/s1600/stress.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="362" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimqhI9GnXt8LS1x4-K4aAzWIFvtirLBeOMXbvfZdQvl8rgIu9Lvo7S0XhEj7Czg_hAmGpsPkHM4iDyaqEJRuZjX0ViHtNejHeveujL11Fe0rHDD9BRcGabROd-Hm-TNaOm1mqv4i_9D-g/s640/stress.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Instances under stress c3 (i-2cb4bd86), default c4 (i-c2b4bd68), tweaked c4 (i-0e737ba4)</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Testing memcached using a simple approach of an infinite loop calling memcslap shows slightly greater differences. Two loops are started in the background to simulate a reasonable load of around 200k packets per second (admittedly on the loopback interface but good enough for now):</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ while true; do memcslap --servers=<server_ip_here>:11211 --concurrency=10 --execute-number=10000 --binary > /dev/null; done &</span></div>
<div>
<br /></div>
<div>
On the c3.8xl sar shows (approximately):</div>
<div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">08:35:51 AM CPU %user %nice %system %iowait %steal %idle</span></div>
<div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">08:38:34 AM all 5.48 0.00 8.61 0.00 0.60 85.31</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 5.50 0.00 4.62 0.00 0.33 89.54</span></div>
</div>
</div>
<div>
<br /></div>
<div>
<div>
The default c4 shows:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">08:35:51 AM CPU %user %nice %system %iowait %steal %idle</span></div>
<div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">08:36:20 AM all 5.01 0.00 0.84 0.00 0.00 94.15</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 5.14 0.00 2.99 0.00 0.00 91.87</span></div>
</div>
</div>
<div>
<br /></div>
<div>
While the tweaked c4 shows:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">08:35:51 AM CPU %user %nice %system %iowait %steal %idle</span></div>
</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">08:36:26 AM all 5.53 0.00 2.12 0.00 0.00 92.35</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 5.26 0.00 2.71 0.00 0.00 92.03</span></div>
</div>
<div>
<br /></div>
<div>
Again not much between them according to sar but CloudWatch shows some larger discrepancies in this case with CPU above 20% for all instances and the default c4 showing between 25% - 30% vs the tweaked c4 (and c3) staying mostly between 20% and 25%.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg28HdizaDN7b2CQZdNSUWnKMBhgO0DvBd76x24v4LeEqnJp755rj7gMuzvgUZToUWvwfjQ3D2tT5s2M9fNqpkeAK4jCwO5nOz-00rP8Pv_3oAQF9GHIfsbPeLlBbch_Loq5QcG9Ra8gaQ/s1600/memcslap.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="352" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg28HdizaDN7b2CQZdNSUWnKMBhgO0DvBd76x24v4LeEqnJp755rj7gMuzvgUZToUWvwfjQ3D2tT5s2M9fNqpkeAK4jCwO5nOz-00rP8Pv_3oAQF9GHIfsbPeLlBbch_Loq5QcG9Ra8gaQ/s640/memcslap.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: 12.8000001907349px;">Instances running memcslap c3 (i-</span>2cb4bd86<span style="font-size: 12.8000001907349px;">), default c4 (i-</span>c2b4bd68<span style="font-size: 12.8000001907349px;">), tweaked c4 (i-0e737ba4)</span></td></tr>
</tbody></table>
<div>
We are clearly missing something important here, time to look at that intel_pstate driver.</div>
<div>
<br /></div>
<div>
<h3>
All about the driver (enabling intel_pstate)</h3>
</div>
<div>
As noted earlier the default cpufreq driver on Ubuntu 14.04 appears to be the legacy acpi-cpufreq driver. There is some history around this which is not really relevant here but suffice it to say that you need to opt-in to enable the intel_pstate driver. Again it is a relatively simple matter of adding a kernel option to grub and given that the evidence above shows our default c4 installation is a loser, lets make some changes to make it faster. Firstly, disable the ondemand scaling policy as we did for the the tweaked c4 earlier, next we will enable the intel_pstate driver by adding 'intel_pstate=enable' to the Grub command line, we are also going to disable the deeper C states at the same time to limit latency of waking cores (with 'intel_idle.max_cstate=1') after which we will install the new Grub config and reboot:</div>
<div>
<br /></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">$ sudo update-rc.d ondemand disable</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">...</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /etc/default/grub.d/50-cloudimg-settings.cfg </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Cloud Image specific Grub settings for Generic Cloud Images</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># CLOUD_IMG: This file was created/modified by the Cloud Image build process</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Set the recordfail timeout</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_RECORDFAIL_TIMEOUT=0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Do not wait on grub prompt</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_TIMEOUT=0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Set the default commandline</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 intel_idle.max_cstate=1 intel_pstate=enable"</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Set the grub console type</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">GRUB_TERMINAL=console</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo update-grub</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">...</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo reboot</span></div>
<div>
<br /></div>
<div>
After the reboot, cpupower confirms that we are now using the intel_pstate driver and that our server is now super-charged at 3.2 GHz:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ sudo cpupower frequency-info</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">analyzing CPU 0:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> driver: intel_pstate</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> CPUs which run at the same hardware frequency: 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> CPUs which need to have their frequency coordinated by software: 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> maximum transition latency: 0.97 ms.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> hardware limits: 1.20 GHz - 3.50 GHz</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> available cpufreq governors: performance, powersave</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> current policy: frequency should be within 1.20 GHz and 3.50 GHz.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> The governor "performance" may decide which speed to use</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> within this range.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> current CPU frequency is 3.20 GHz (asserted by call to hardware).</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> boost state support:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Supported: yes</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Active: yes</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">$ cat /proc/cpuinfo | grep MHz</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">[...]</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">cpu MHz</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; font-size: xx-small; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">cpu MHz<span class="Apple-tab-span" style="white-space: pre;"> </span>: 3200.875</span></div>
</div>
<div>
<br /></div>
<div>
Rerunning the earlier memcached test shows quite a significant improvement for the P-state instance. While the sar results are quite similar, the CloudWatch graph shows the P-state instance using around 16% CPU, beating both the c3 and tweaked c4. The sar approximate average for the P-state c4:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">09:09:52 AM CPU %user %nice %system %iowait %steal %idle</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">09:10:36 AM all 7.17 0.00 3.75 0.00 0.00 89.08</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Average: all 6.15 0.00 2.43 0.00 0.00 91.42</span></div>
</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeXhktaJDSJ66HOlaCvVgSnhLyrSIzebwzQWFLjw1_9HqWi0Kb9-I-gcdfskdSEANxUlMlm4lwlb-injQmfiPaqhwzgd4hcZcsVnNI-56YpjgGRjFPP9stM3tdtR8U2lJhlBjQIAEGaFY/s1600/p_state.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="352" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeXhktaJDSJ66HOlaCvVgSnhLyrSIzebwzQWFLjw1_9HqWi0Kb9-I-gcdfskdSEANxUlMlm4lwlb-injQmfiPaqhwzgd4hcZcsVnNI-56YpjgGRjFPP9stM3tdtR8U2lJhlBjQIAEGaFY/s640/p_state.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Instance<span style="font-size: 12.8000001907349px;">s running memcslap c3 (i-</span>2cb4bd86<span style="font-size: 12.8000001907349px;">), intel_pstate c4 (i-c2b4bd68), tweaked c4 (i-0e737ba4)</span></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
For comparison rerunning the stress test from earlier (stress -c 4 -m 2):</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipJ1dGHGxoWrLJwV6xjwRxVUQGMKUzMM-P95Z4MNtLGuXy-39Bpg-nz_i5C9qRO3glZzI0zyGfJQPmlwU0FcIWYbIYl_eF6yj_8E7P_5avZGfdmv2T20mJm6wPQjiLMZ0b7kASk5ipzko/s1600/p-state_stress.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="352" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipJ1dGHGxoWrLJwV6xjwRxVUQGMKUzMM-P95Z4MNtLGuXy-39Bpg-nz_i5C9qRO3glZzI0zyGfJQPmlwU0FcIWYbIYl_eF6yj_8E7P_5avZGfdmv2T20mJm6wPQjiLMZ0b7kASk5ipzko/s640/p-state_stress.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: 12.8000001907349px;">Instances under stress c3 (i-2cb4bd86), intel_pstate c4 (i-c2b4bd68), tweaked c4 (i-0e737ba4)</span></td></tr>
</tbody></table>
<div>
<br /></div>
<h3>
Summary</h3>
<div>
After a relatively quick and dirty investigation it is clear that the intel_pstate driver while not enabled by default on Ubuntu 14.04 really should be the driver of choice.</div>
<div>
<br /></div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-69013513391962036672015-06-22T14:35:00.001+01:002015-06-22T14:35:48.660+01:00AWS Tip: Terminating instances in an Auto Scaling groupEC2 instances launched by Auto Scaling really should not be terminated outside of Auto Scaling but if for some reason you (really, really) need to terminate a number of instances you may see some unexpected behaviour if you terminate them using the EC2 console. Instead of all the instances being replaced by Auto Scaling simultaneously they are replaced individually over a period of time. This means that it may take significantly longer than you would expect to get the ASG back to the original size. As an <a href="https://gist.github.com/watchamcb/fc686c70e8c2660ee7ef" target="_blank">example</a> an instance in my test ASG was replaced every 2 minutes, taking around 6 minutes to terminate and replace 3 instances.<br />
<br />
There are two ways of doing this more efficiently (using the supremely useful <a href="http://docs.aws.amazon.com/cli/latest/index.html" target="_blank">AWS CLI</a>), firstly by setting the group capacity and then resetting it:<br />
<br />
<br />
<script src="https://gist.github.com/watchamcb/f798d361a8b74fb0a707.js"></script>
Note that <a href="http://docs.aws.amazon.com/cli/latest/reference/autoscaling/update-auto-scaling-group.html" target="_blank">update-auto-scaling-group</a> is used due to the minimum size being set, if no minimum size is specified the same result can be achieved with <a href="http://docs.aws.amazon.com/cli/latest/reference/autoscaling/set-desired-capacity.html" target="_blank">set-desired-capacity</a>.<br />
<br />
The second option is to use <a href="http://docs.aws.amazon.com/cli/latest/reference/autoscaling/terminate-instance-in-auto-scaling-group.html" target="_blank">terminate-instance-in-auto-scaling-group</a>:<br />
<br />
<br />
<script src="https://gist.github.com/watchamcb/cf6e8e4a03f204fe9fe4.js"></script>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-88204302980952813612015-01-19T10:37:00.002+00:002015-03-07T04:46:23.373+00:00AWS Tip of the day: Working around the S3 "Failed to parse XML document" exceptionIf you have S3 objects with Unicode characters that aren't supported by XML 1.0 you are likely to see an exception when calling <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#listObjects(com.amazonaws.services.s3.model.ListObjectsRequest)" target="_blank">listObjects</a> in the AWS Java SDK:<br />
<div>
<br /></div>
Failed to parse XML document with handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
<div>
<br /></div>
<div>
The quick and simple solution to fix this is to set the <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html#withPrefix(java.lang.String)" target="_blank">encoding type</a> to "url". Example code stolen from <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysUsingAPIs.html" target="_blank">here</a>:</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/watchamcb/5d1c92b3acb56a34b9ec.js"></script>
</div>Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-70311909536418211892014-10-05T09:11:00.000+01:002014-10-05T09:11:05.240+01:00AWS Tip of the day: Signing requests using IAM instance rolesIf you are averse to having the AWS SDKs conveniently manage <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html" target="_blank">IAM instance role</a> permissions and service requests (including <a href="http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html" target="_blank">Signature V4</a>) then knowing that you need to include the IAM role token in the request header is quite important. This post describes the steps needed to sign an API request using an instance IAM role. It is assumed that you already have an instance launched with a role that has permission to perform the requested API action. For this example the role is going to be named ec2-ro and as the name implies it has read-only permissions on EC2 APIs.<br />
<br />
As a good starting point we will use the first GET example on <a href="http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html" target="_blank">this</a> page as a working base and modify it to retrieve the instance role credentials and add the session token to the request as follows.<br />
<br />
<h4>
Step 1: Retrieve the instance role credentials</h4>
As per the AWS <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials" target="_blank">documentation</a>, instance credentials can be retrieved from the instance metadata by performing a GET against the following URL, where <role-name> is the name of the instance role containing the relevant permissions:<br />
<br />
http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name><br />
<br />
In our example with the role named ec2-ro the result of the request will be similar to the result below where the body of the token has been replaced with [...] for brevity:<br />
<br />
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-ro<br />
{<br />
"Code" : "Success",<br />
"LastUpdated" : "2014-10-05T07:25:22Z",<br />
"Type" : "AWS-HMAC",<br />
"AccessKeyId" : "ASIXXXXXXXXXXXX",<br />
"SecretAccessKey" : "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234",<br />
"Token" : "AXXXXXX//////////[...]==",<br />
"Expiration" : "2014-10-05T13:30:35Z"<br />
}<br />
<br />
Achieving the same in Python is pretty simple:<br />
<br />
creds = requests.get('http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-ro')<br />
access_key = creds.json()['AccessKeyId']<br />
secret_key = creds.json()['SecretAccessKey']<br />
token = creds.json()['Token']<br />
<div>
<br /></div>
Which we can then use to replace access key code in the example:<br />
<br />
access_key = os.environ.get('AWS_ACCESS_KEY_ID')<br />
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')<br />
<br />
Unfortunately we are not finished yet as running the new code now results in a 401 response with the following message:<br />
<br />
"AWS was not able to validate the provided access credentials"<br />
<br />
The reason for the error above is that the instance credentials require a session token to be considered valid, moving on to the next step.<br />
<br />
<h4>
Step 2: Add the session token to the request headers</h4>
Fixing the invalid credential error above is relatively trivial and simply involves adding the session token to the request headers with a header name of x-amz-security-token. So replacing this line:<br />
<br />
headers = {'x-amz-date':amzdate, 'Authorization':authorization_header}<br />
<br />
With the following line:<br />
<br />
<span id="caseCorrespondence_27425114255_text">headers = {'x-amz-date':amzdate, 'Authorization':authorization_header, 'x-amz-security-token':token}</span><br />
<span><br /></span>
<span>Allows the code to execute successfully. </span>One thing to be aware of is that the instance credentials are rotated fairly frequently and the credentials will need to be refreshed after the date and time in the instance metadata Expiration field.Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-57806081467133261432014-09-14T10:55:00.002+01:002014-09-14T10:55:43.169+01:00AWS script of the day: Core countIn case you are curious or just want to know how close you are to being a super computer, here is a quick script to count the number of cores an AWS account is currently running: <a href="https://github.com/watchamcb/aws/blob/master/core_count.py" target="_blank">core_count.py</a><br />
<br />
Make sure you have <a href="https://boto.readthedocs.org/en/latest/" target="_blank">boto</a> setup up first.Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-35184300277420949472014-09-13T14:19:00.002+01:002014-09-13T14:19:54.901+01:00AWS script of the day: Cascade delete of security groupsA common bug bear of AWS security groups is having to delete all references to a security group before deleting the group itself. <a href="https://github.com/watchamcb/aws/blob/master/sg_cascade_delete.py" target="_blank">Here</a> is a quick boto script to simplify this process, you will need to have configured boto as per these <a href="http://docs.pythonboto.org/en/latest/getting_started.html" target="_blank">instructions</a>. After which 'python sg_cascade_delete -h' will give you:<br />
<br />
usage: sg_cascade_delete.py [-h] [--region REGION] [--quick] [--force]<br />
[--quiet]<br />
group_ids [group_ids ...]<br />
<br />
Remove all references to a security group and then delete it<br />
<br />
positional arguments:<br />
group_ids The ID of the security group to delete, eg. sg-xxxxxxx<br />
<br />
optional arguments:<br />
-h, --help show this help message and exit<br />
--region REGION AWS region name the security group is in, default: us-<br />
east-1<br />
--quick Skip checks for whether or not the group is used by<br />
RDS/ElastiCache. Faster but may cause error on delete if<br />
the group is referenced.<br />
--force Force delete without requiring confirmation<br />
--quiet Do not print references or success message<br />
<div>
<br /></div>
<div>
<br /></div>
<div>
An example of usage would be:</div>
<div>
python sg_cascade_delete --region eu-west-1 sg-1231234</div>
<div>
<br /></div>
<div>
This will find all references to the sg-1231234 security group in the region and display them before asking for confirmation to delete the group. Note that you will be prevented from deleting any groups used in ElastiCache or RDS security groups as doing so tends to break things in unexpected ways.</div>
<div>
<br /></div>
<div>
If you don't want to have to confirm the deletion (for a large number of groups for example) you can specify the --force option, this will skip the confirmation question and simply delete the groups after displaying their references. For example:</div>
<div>
python sg_cascade_delete --force --region eu-west-1 sg-1231234 sg-33221133</div>
<div>
<br /></div>
<div>
If you prefer your deletion silent then the --quiet option is for you, specifying this will prevent any messages being printed (other than the confirmation question and errors that occur). For no interaction at all use with --force to magically delete the groups without a sound. A non-zero process exit code indicates an error.</div>
<div>
<br /></div>
<div>
If you have a large number of ElastiCache clusters and RDS instances you can skip the reference checks by specifying the --quick option, this may result in errors (in VPC) if the group is actually referenced when trying to delete and will cause some strange behaviour in EC2 classic as you are actually able to delete the group leaving a dangling authorisation on the ElastiCache/RDS security group. As such it is advised that you use this option with care or when you are truly certain that the security group is not used anywhere but in EC2.</div>
<div>
<br /></div>
<div>
As this code is mutating (it changes your stuff) it would be wise to run it in a test environment before making changes in production. In other words: use at your own risk.</div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-84818983785862942432014-09-03T09:06:00.004+01:002014-09-03T09:06:45.308+01:00AWS Tip: Upgrade your PHP 1.x SDK<span>If you are still using 1.6.2 of the <a href="https://github.com/amazonwebservices/aws-sdk-for-php" target="_blank">AWS PHP SDK</a> you can expect some strange issues, for example:</span><br />
<span><br /></span>
<span id="caseCorrespondence_27233513465_text">$ec2->describe_instances();</span><br />
<span><br /></span>
<span>Will not list T2 instance types. Save yourself some hassle and upgrade to the <a href="https://github.com/aws/aws-sdk-php" target="_blank">latest</a> version with the help of this <a href="http://docs.aws.amazon.com/aws-sdk-php/guide/latest/migration-guide.html" target="_blank">migration guide</a>.</span><br />
<br />Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-78280972278410723432014-08-30T09:56:00.002+01:002014-08-30T09:56:45.564+01:00AWS Tip: elmo.ec2sg preventing security group deletion<span>If you are trying to delete an EC2 security group and get an error something like:</span><br />
<span><br /></span>
<span id="caseCorrespondence_27243625145_text">sg-12345678: Group 111222333444:My SG Name is used by groups: 123412341234:elmo.ec2sg.887766</span><br />
<span><br /></span>
<span>Save yourself some time and check that your ElastiCache and RDS security groups don't reference (by name) the group you are trying to delete. </span>See <a href="https://forums.aws.amazon.com/thread.jspa?messageID=555154" target="_blank">this</a> forum thread.Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.comtag:blogger.com,1999:blog-8082302445306001355.post-64755537873477509092014-06-28T13:09:00.000+01:002014-06-28T13:09:25.584+01:00Installing the AWS .NET SDK on Mono/Linux using NuGetFollowing a previous <a href="http://www.deplication.net/2014/04/aws-net-sdk-on-monolinux.html">post</a> about compiling the AWS .NET SDK on Mono/Linux <a href="https://plus.google.com/114372976812620049955">+David Fevre</a> asked whether it was possible to use <a href="http://www.nuget.org/">NuGet</a> to install the SDK. It turns out that this is actually quite a bit simpler if you are running Mono 3.2+. Below are the steps required to get the AWS SDK working on Mono/Linux (Ubuntu 14.04)<br /><br />1. Install mono and associated tools, mozroots downloads root certificates to enable SSL via mono:<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">sudo apt-get update<br />sudo apt-get install mono-complete git wget<br />mozroots --import --sync</span><br /><br />2. Create a working directory and change to it:<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">mkdir mono-aws<br />cd mono-aws</span><br />3. Download the command line version of NuGet and verify that it is version 2.8+:<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">wget http://nuget.org/nuget.exe<br />mono nuget.exe update -self</span><br /><br /> 4. Install the AWSSDK via NuGet:<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">mono nuget.exe install AWSSDK</span><div>
<br /></div>
<div>
5. Create a symbolic link to the lib folder (note that the version number might be different from the example command):</div>
<div>
<span style="font-family: Verdana, sans-serif; font-size: x-small;">ln -s AWSSDK.2.1.6.0/lib/net35 lib</span><br /><br />6. Create a command line program to test with, for example in s3.cs:<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">using System;<br />using Amazon.S3;<br />using Amazon.S3.Model;<br /><br />class Upload<br />{<br /> public static void Main(string[] args)<br /> {<br /> // Create a client<br /> AmazonS3Client client = new AmazonS3Client(Amazon.RegionEndpoint.USWest2);<br /><br /> // Create a PutObject request<br /> PutObjectRequest request = new PutObjectRequest<br /> {<br /> BucketName = "SampleBucket",<br /> Key = "Item1",<br /> ContentBody = "This is sample content..."<br /> };<br /><br /> // Put object<br /> client.PutObject(request);<br /> }<br />}</span><br /><br />7. Compile your program:<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">mcs s3.cs -r:./lib/AWSSDK.dll</span><br /><br />8. Set the Mono path<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">export MONO_PATH=`pwd`/lib:.</span><br /><br />9. Create an application configuration (must be named to match the compiled program) for example in <span style="font-family: Verdana, sans-serif; font-size: x-small;">s3.exe.config:<br /><?xml version="1.0"?><br /><configuration><br /> <appSettings><br /> <add key="AWSAccessKey" value="AKXXXXXXXXXXXXXX"/><br /> <add key="AWSSecretKey" value="XXXXXXXXXXXXXXXXXXXXXXXX"/><br /> </appSettings><br /></configuration></span><br /><br />10. Run your program and verify that the object is created in the bucket:<br /><span style="font-family: Verdana, sans-serif; font-size: x-small;">mono s3.exe</span></div>
Craig Watchamhttp://www.blogger.com/profile/16798624121656993806noreply@blogger.com