re:Invent 2019

Summary

  • AWS Outpost are Generally Available for Ordering – your customers can now get AWS physical infrastructure on-premise
  • End-of-Support Migration Programme for Windows – lets customers move applications off older versions of Windows without breaking them
  • Network Manager – enables a holistic view of all connectivity to AWS including on-premise VPNs, DirectConnect and SD-WAN deployments based on Cisco, Aruba, Silver Peak, and Aviatrix
  • AWS Wavelength – puts AWS resources directly into mobile carrier networks to improve the application experience for mobile users
  • IAM Access Policy Manager – helps audit and secure the access policies assigned to resources such as S3

About re:Invent

AWS held its first conference in 2012 in Las Vegas, and here in 2019 it’s back again with an attendance in excess of 65,000 and taking over the conference centres of six of Las Vegas’ biggest hotels. Starting from Monday and running through to Friday there are over 3000 scheduled sessions with more added as the week progresses.  This is a technical conference through and through – while there’s a splatter of marketing here, the majority of content is aimed at those building services and applications with AWS, as well as leadership around how to adapt enterprises and their IT operations to public cloud.

AWS in the Public Cloud Market

AWS are very proud (and boastful) of their public-cloud market share – holding 47% of the market, with Azure following at 15%, Alibaba at 7% and GCP 4%. The remaining 27% is split across Oracle, IBM and a few other niche platforms. Already, AWS’ first two quarters of revenue in 2019 have exceeded 2018’s entire reported revenue – they’re seeing 39% growth predictions for this year!

New Product Launches

AWS Outposts End-of-Support Migration Programme for Windows
First announced last year, has finally gone “GA” or Generally Available – this means that customers can order pre-built AWS compute infrastructure to be installed in their own on-premise DCs.  Designed to solve some of the issues around application latency and data locality – the infrastructure is managed and controlled right from the AWS console and is an extension of a customer’s AWS environment. One of the biggest launches this week is a migration service that essentially wraps applications that are dependent on older generations of Windows (2003, 2008 etc) in an envelope and allows you to port them onto newer Windows editions.  This means you can take advantage of the critical security and performance updates available in newer versions of Windows.

Other Notable New Services

Network Manager Introduces a new way to manage and monitor your use of AWS’ global networks as well as how it interfaces with your on-premise networks via Site-to-Site VPN and SD-WAN.   CiscoAruba, Silver Peak, and Aviatrix have all announced integrations of their SD-WAN products with Network Manager.
Amazon VPC Ingress Routing You can now segment your Amazon Virtual Private Cloud traffic so that it is routed via virtual appliances, both inbound and outbound. 
Access Analyzer for S3 and IAM Access Analyzer These new features monitor access policies and enables proactive remediation of potentially unwanted access.  
AWS License Manager additional functionality Dedicated hosts can be difficult to manage for certain licensing considerations (for example BYOL).  AWS License Manager now simplifies this.
Wavelength Will bring AWS services and capabilities as close to mobile users as possible by putting AWS resources directly in 5G carrier network hubs.
AWS Data Exchange AWS already have MarketPlace for ISVs and pre-built solutions – Data Exchange allows companies to share / sell data which might be useful to others. Examples of this include anonymised healthcare insights  or historic news items.
Amazon Bracket Probably not relevant to 99.99% of our customers but AWS have brought Quantum Computing to the cloud.

Networking

Networking is key to how AWS provides its Virtual Private Cloud (VPC) – enabling it to not only host virtual machines (AWS calls them Instances) in its infrastructure but to connect those VMs with the Internet and on-premise networks.  Here’s a couple of updates from the VPC world:

  • AWS Transit Gateway Multicast
    • Multicast, in the cloud… used most often by media broadcasters and financial/energy trading customers, the lack of this would have once been a show-stopper for cloud-adoption.
  • Accelerated Site-to-Site VPN
    • Brings the VPN gateway to an edge location closest to your on-premise VPN connection
    • Used in conjunction with transit gateways
    • Uses AWS backbone network and is essentially driven using anycast connectivity
  • AWS Transit Gateway Inter-Region Peering
    • Now you can connect between VPCs in different regions – previously you would have to do this with site-to-site/IPSEC tunnels.

Under The Hood

re:Invent isn’t just about learning how to use AWS’ technologies and services, it’s also about learning about what goes on under the hood (or behind the silver lining):

  • Graviton 2 ARM Chips – the next generation of ARM-powered Instances (this differs from your typical Intel/AMD x86 instance types)
  • Nitro 2 Controller – AWS use a specialised and custom built virtualisation controller called “Nitro” – this week saw the confirmation of its second generation being employed, providing low-latency 100Gbps network connectivity to convince the high-performance compute crowd that you can do HPC in the Cloud  (compared to 25Gbps previously).

Building a new AWS Lab (part 2) – Billing

I mentioned in Part 1 that I wasn’t sure of the costs of doing a lab environment with Control Tower (plus all the resources it turns on by default). To keep on top of things, I’ve enabled billing notifications with a monthly $20 limit – this needs to be done in the original route account (so logging into AWS with your root credentials and NOT the SSO credentials).

I’ll come back to this in a week and see how the bill is working out..


Building a new AWS Lab (part 1.5)

Preventing Sprawl!

Since this is only and lab and I want to keep the costs under control, I’ve opted to set some lower retention/lifecycle policies in the buckets related to logs (the default is 365 days).

Since ControlTower deploys everything using StackSets, you need to modify the StackSet rather than editing the S3 Lifecycle policy directly.

Login as your AWS Administrator from your SSO console and go to Shared Accounts -> Log Archive and click on View CloudFormation StackSet:

Now you can modify the Retention policy in the Stack parameters by clicking through:

  • Manage StackSet
  • Edit StackSet
  • Current template: Update AWSControlTowerLoggingResources
  • and then changing the retention policy to 5-days.

After this, just Next-Next-Next until you can update the StackSet… this will rollout to the account automatically.


Building a new AWS Lab (part 1)

So, I’m deep in the midst of my AWS SA Pro studies this week but need to put some things into practice to help cement the ideas.. to do this I’ve decided to build a new AWS environment using all the best/recommended/guided methods I could!

First steps today – getting a basic account structure up and running using AWS Control Tower.  Control tower builds a basic account structure called a Landing Zone – consisting of the root account, a logging account and an auditing account, and then wrapping this in a set of permissions, with SSO and an Organisations setup.

Warning Side note: this is probably going to cost a fortune to keep running, but it’s only for a short period.. I don’t think the Free Tier will cover much.  I’ll try and do a costs-related post later on once they start racking up.

Steps to do first thing:

  1. Create basic AWS account – I’ve done this, given it a crazy lab-related name and setup a dedicate email alias in Gmail for the first tranche of emails.
  2. Created a WorkMail configuration with two email addresses; one for notifications (logs) and one for auditor (audit logs)
  3. Kicked off the Control Tower build process

WorkMail

WorkMail is quite useful here as I can completely contain all the email etc associated with the Organisation/LandingZone within the original account.  I wouldn’t recommend this as best-practice, but it makes it easy to dismantle the lab environment afterwards.

Costs…  (verbatim) $4.00 per user per month and includes 50 GB of mailbox storage for each user. You can get started with a 30-day free trial for up to 25 users.

Control Tower

Control Tower delivers a basic level of security and governance around AWS accounts.. getting the customer off to the right start on the Shared Responsibility Model.

As Control Tower sets up SSO, you’ll get an email to setup your SSO account – once done you should be able to login and get a view similar to this:

First SSO Account View

It takes about an hour for Control Tower to do its thing and the dashboard will keep you updated with its progress:

And once it’s done, something like this:


Learning: NoSQL and DynamoDB

As part of my AWS-SA-Pro studies, I’m finding myself doing a lot of reading / watching on the subject of NoSQL databases and in doing so have come across a video from re:Invent 2018 presented by Rick Houlihan (Principle Technologist, NoSQL).  This one slide pretty much sums up NoSQL for me:

Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401)

Tables in NoSQL (specifically DynamoDB) have just a few things that you can base your query on; within the Primary Key there is a partition key and sort key.  You can create relationships by using the partition key as a grouping mechanism and the sort key as the most common attribute from which you want to query those relationships.

You can create further sorting keys from other data using Global and Local Search Indexes (GSI, LSI) – but these effectively create copies of the table data organised with alternative fields forming the Primary Key, so you want to minimise the number of GSI and LSI you use.

GSI_Slide

Global Secondary Index – effectively uses a completely different field to organise the data with

LSI_Slide

Local Secondary Index – maintains the original partition key (original grouping attribute) but enables a different attribute to sort from.

The whole point of NoSQL is to create a scalable database which reduces the load on the CPU that’s inherent with complex relational databases.. taking something like a delivery service, with 6 different tables in SQL and reducing it down to one table and three GSI in NoSQL.  The table format and attributes you use for your Primary and Sort keys all depend on the access methods patterns – ie, what sort of queries you run on the data (what data are you looking for and what’s your input variables/fields for retrieving that data).

RDSApproach-DeliveryNoSQL-Approach_Delivery


AWS Connect

I started my networking career when I joined Cisco as an intern back in 2006 – and my first few projects rotated around CUCM. Later in life I did a project for Vodafone for a hosted Contact Centre service.  At the time, the amount of effort, cost and complexity that went into buildinging these solutions seemed appropriate – seemed normal.  A few years later and I now think it’s horrifying!

Two hours.  That’s all it took for me to learn the basics of AWS Connect and build a basic call flow with some voice-recognition using Lex.  Two hours – and I’d never touched AWS Connect or Lex before.

Read the rest of this entry »


AWS Exam Review

Last week I took and passed the AWS Solutions Architect Associate exam.  Other than the rather weird PSI test centre/environment/process (more on that later) I actually rather enjoy this examination.

Don’t worry – there’s no NDA-breaching here.

The exam itself presents 65 questions with just over two-hours to complete.  Most of the questions are scenario based, rather than quick-win factoid answers.  This is why I liked it.  I’m terrible at remember factoids (ie, this thing costs $0.065/MB on a Wednesday when the sun is shining and the wind is from the north).  But I am very good at putting things into practical use and designing around capabilities or intended functionality.  What’s the best way to ensure maximum availability for static content used on customer X’s website?  What’s the best way to provide maximum fault tolerance across availability zone’s when an application needs Y instances.    These are all great questions that make you think about the capabilities of AWS’s services, how they are distributed and how you can exploit them.

I used a number of study aids in preparing for the exam.

Firstly; A Cloud Guru – has to be the best technical training platform I’ve ever encountered.  The courses are delivered in chunks of 10~20 mins, so you’re never overloaded, and the majority have a practical element that you can follow-along with.  The quizzes at the end of each module keep you on track and the exam simulations are a realistic enough to give you a view of your progress.

Secondly; read the whitepapers. I can’t stress this enough – the baseline should be a thorough read of the Well-Architected Framework and the Cloud Adoption Framework.  You should also reading AWS blogs and design on solutions that have already been architected.

Finally; the AWS Free Tier.  Exploit it! Tinker with everything.  The more you get a practical view of AWS the more you understand how things hook together and where the limitations are (or aren’t).

The last point I have is around the PSI test experience.  I’m very used to doing Pearson Vue exams – the way the test centres work and how the exams work, be the Cisco or VMware or whatever – they are very structured.  The PSI experience was very lightweight.  I was given a username and one-time-password to use at the exam machine and that was it.  When the test is finished, you don’t get a score report (displayed or printed), you just get a “Congratulations” on an “Unsuccessful”.  The screen then kinda leaves you hanging here – I recommend calling for a proctor at this point and get them to log the exam out.

Next up.. AWS SysOps Associate!


Pirate-Jim Here We Come ’17

Amongst the diving and working it seems this year I’ll also be partaking in some sailing too..

  • May – Quick whiz around the Solent with Commodore Yachting to reacquaint myself with a yacht
  • June – Sailing around Norway from Bergen to Aalesund
  • November – All being well, I may be sailing the Atlantic Rally for Cruisers from Las Palmas (Spain) to St Lucia

Simple AWS Diagrams

So my most recent post included some Visio-style diagrams but not done in Visio.. try draw.io out, it’s pretty good. Basic, but good.


My First vPC

my first vpc

My first attempt at building some form of basic infrastructure constructs in AWS.. Keep in mind that this is the learning curve, so in no way represents best-practice deployments!

The Building Blocks

  • Internet Gateway
  • Single vPC in London Region (eu-west)
  • Two subnets, one in each availability zone (eu-west-2a and eu-west-2b) for Web Servers
  • Two subnets, one in each availability zone for Bastion hosts
  • Two Launch Configurations – one for bastion hosts, one for webservers
  • One Auto Scaling Group for Web Servers – min instances 2, linked to webserver Launch Configuration
  • One Auto Scaling Group for Bastion host – min instances 1, linked to Bastion Launch Configuration
  • Elastic Load Balancer – inbound  HTTP/s connected to the Web Server auto-scaling group
  • One Security Group for Web Servers – enables inbound HTTP / HTTPs from anywhere, and SSH from the Bastion Subnets
  • One Security Group for Bastion hosts – enables inbound SSH from anywhere
  • One IAM Role – to enable Read-Only access to S3

Web Server Launch Configuration

Each web server is built using a Launch configuration which has a bootstrap script to do the following:

  • Update standard AMI packages
  • Install Apache and PHP
  • Start Apache
  • Set Apache to start on bootup
  • Copy custom index.php from S3  (this is why it needs an IAM role to access S3!)
  • Copy health-check HTML from S3
  • Make index.php executable

The index.php is a basic “Hello World” which also shows the internal IP of the host serving it.. this way when tweaking with load-balancers I can tell which instance has served the request.  The two pages are stored in an S3 bucket and the IAM role applied to the Launch Configuration allows the instances to copy the files down to the web server.

 #! /bin/bash
 yum update -y
 yum install http php -y
 service httpd start
 chkconfig httpd on
 aws s3 --region eu-west-2 cp s3://e02-lab-scripts/index.php /var/www/html/
 aws s3 --region eu-west-2 cp s3://e02-lab-scripts/healthcheck.html /var/www/html/
 chmod +x /var/www/html/index.php

Summary

This stuff is bloody complicated – but – certainly not impossible.  Once you know what all the components are, how they work and interact with each other, it’s easy to start building services and constructs based on them.


Bitnami