vRA Data Center on Demand with Terraform

Don’t you just hate it when you get that email statement for your monthly AWS bill and you realized you forgot to shutdown your latest experiment. Just like I have automated the start/stop of the homelab, I decided it was time to extend the functionality to AWS setup and vRA configuration. The project gave me an excuse to learn me some Terraform and give the vRA Terraform provider a try as well. This was my first time using Terraform and i may have been a bit ambitious for my first use case but I learned a lot.

Design goal is a fully automated AWS VPC spanning multiple availability zones, subnets, security groups, internet gateway, route tables. I also required the vRA constructs  automated including the Cloud Account, Cloud Zones, Network Profiles, Image and Flavor Mappings. I could have used AWS CloudFormation and vRA API, but Terraform provided the tool to manage both stacks from a common configuration file without having to resort to coding against the API’s myself.

First off is the AWS VPC design. Using Terraform makes this an easy iterative process. On my way to a three tier web app leveraging application load balancers with pubic/private subnets. But for now only have 2 tiers defined and both open to the internet for easy testing and validation. Forgive me for poor security practices, but I will be tightening up in future iterations of the environment.


AWS VPC Diagram

I spent extra time figuring out how to dynamically add/remove availability zones using a  variable map to define the target zones. The az variable then drives the subnet creation with automated names, ranges, routes and security group members. Lots of experimentation for a Terraform newbie to figure out loops, functions and data references.

Variable: subnet_numbers mapping

## Availablility zones for VPC
variable "subnet_numbers" {
  description ="Map availability zone to subnet numbers"
  default = {
    "us-east-1a" = 1
    "us-east-1b" = 2
#   "us-east-1c" = 3
  }
}

The resource to dynamically create the web subnets using the availability zone mapping.

##  Web subnets
resource "aws_subnet" "web_subnets" {

  # iterate thru the availability zones to subnet number mapping
  for_each = var.subnet_numbers

  vpc_id     = aws_vpc.webapp-vpc.id

  # Automatically caluculate the subnet from the VPC cidr and mapping

  cidr_block = cidrsubnet(aws_vpc.webapp-vpc.cidr_block,8,each.value)

  availability_zone = each.key

  tags = {

    # used the subsring on naming to extract the last 
    # 2 chars of az for name spec (ie: web-1a)

    Name = join ("-",["web",substr(each.key,8,9)])

    Application="webapp"
    Tier="web"
    Index=each.value
  }
}

Subnets in the AWS Console


AWS Subnets

I used similar methods for route associations end security groups.

### Route tables and associations
resource "aws_route_table" "webapp-rt" {
  vpc_id= aws_vpc.webapp-vpc.id
  route {
    cidr_block="0.0.0.0/0"
    gateway_id=aws_internet_gateway.webapp-igw.id
  }
  tags = {
    Name = "webapp-rt"
  }
}

resource "aws_route_table_association" "web_routes" {
  # Iterate through the web subnets to add to route
  for_each = aws_subnet.web_subnets
  subnet_id=each.value["id"]
  route_table_id=aws_route_table.webapp-rt.id
}

Now that AWS is configured its time to start configuring vRA infrastructure constructs. First is to get the terraform-provider-vra installed and configured available on github.  The VMware blog “Getting started with the vRealize Automaton Terraform Provider” by Sam McGeown is a great resource where I started to figure this all out.

Setting up the cloud account was pretty straight forward, but I also needed to configure my cloud zones dynamically. You have to use specific data blocks to retrieve vRA construct identifiers to configure resources as well. The cloud zones require the vra_region to be retrieved from vRA.  Now it’s back to iterating the subnets/availability zone variable to create multiple cloud zones.

# Get the vra_region to create the Cloud Zones
data "vra_region" "lab" {
  cloud_account_id = vra_cloud_account_aws.lab.id
  region = var.region
}

# Configure a new Cloud Zone
resource "vra_zone" "aws" {
  for_each = var.subnet_numbers
  name = join(" ", ["AWS",each.key])
  
  # generate cloud zone description from the AZ
  description = join(" ", ["Cloud Zone configured by Terraform",each.key])
  
  region_id = data.vra_region.lab.id

  tags {
    key = "zone"
    value = each.key
  }
}

Flavor and Image mappings setup. I attempted to iterate through a variable mapping for the flavor name/type would only create the first one and then fail so went back to simple hard coding as these are really pretty static.

resource "vra_flavor_profile" "lab" {
  name = "terraform-flavor-profile"
  description = "Flavor profile created by Terraform"

  region_id = data.vra_region.lab.id
  flavor_mapping {
    name =          "small"
    instance_type = "t3a.nano"
  }
  flavor_mapping {
    name =          "medium"
    instance_type = "t3a.micro"
  }
  flavor_mapping {
    name =          "large"
    instance_type = "t3a.small"
  }
}

# Create a new image profile
resource "vra_image_profile" "lab" { 
  name = "terraform-aws-image-profile"
  description = "AWS image profile created by Terraform"

  region_id = data.vra_region.lab.id
  image_mapping {
    name =      "docker"
    image_name = var.ami
  }
}

Last are the network profiles which are being configured but these are a bit troublesome. I am having issues getting security groups configured properly on the profiles and on every Terraform run the profiles get updated even if no change is expected. I have not dug through the issues on the provider to determine if its a provider problem or just me.

To create the network profile the provider needs a data block to query the networks discovered by vRA for the cloud account. A data block is again using the subnets variable to scan and dynamically get the list of networks to add to hte profiles.

data "vra_fabric_network" "web" {
  for_each=aws_subnet.web_subnets
  #  Iterate the subnets and extract the Name from tags to filter
  filter = "name eq '${each.value.tags.Name}'"
  depends_on = [vra_cloud_account_aws.lab]
}

resource  "vra_network_profile" "web" {
  name = "aws-web"
  description = "AWS Web Tier Profile"
  region_id=data.vra_region.lab.id
  
# Iterate loop to get the list of fabric ids to add to the profile

  fabric_network_ids = [for i in data.vra_fabric_network.web: i.id]

# This is not working. ID is extracted and shows in the Terraform update
# but in vRA gui security group is not associated.

  security_group_ids=[aws_security_group.web-sg.id]

  isolation_type="NONE"
# Add constraints to the profile

  tags {
    key="Tier"
    value="web"
  }
}

When I run “terraform apply” I get 21 configurations created but a few errors on the vRA network profiles. This is a timing issue in vRA where the cloud account has been added, but the networks had not been discovered yet. Still working on how to fix in the Terraform code, but simply running the apply again completes the configuration. Added a provisioner local-exec with a simple 10 second sleep and got passed this issue.

terraform apply

Plan: 21 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_vpc.webapp-vpc: Creating...
aws_vpc.webapp-vpc: Creation complete after 6s [id=vpc-04d6e6ca0ebd042e5]
aws_internet_gateway.webapp-igw: Creating...
aws_subnet.app_subnets["us-east-1a"]: Creating...
aws_subnet.app_subnets["us-east-1b"]: Creating...
aws_subnet.web_subnets["us-east-1b"]: Creating...
aws_subnet.web_subnets["us-east-1a"]: Creating...
aws_security_group.web-sg: Creating...
aws_subnet.web_subnets["us-east-1b"]: Creation complete after 2s [id=subnet-0b31ac3cb3815cbbc]
aws_subnet.app_subnets["us-east-1a"]: Creation complete after 2s [id=subnet-0b6354e438320e629]
aws_subnet.app_subnets["us-east-1b"]: Creation complete after 2s [id=subnet-0e7872f941509749b]
aws_security_group.db-sg: Creating...
aws_subnet.web_subnets["us-east-1a"]: Creation complete after 2s [id=subnet-02ff969e1f81f6dfa]
aws_security_group.app-sg: Creating...
aws_internet_gateway.webapp-igw: Creation complete after 2s [id=igw-0d57829c0395c4191]
aws_route_table.webapp-rt: Creating...
aws_security_group.web-sg: Creation complete after 4s [id=sg-08c234ed2639b13bf]
aws_route_table.webapp-rt: Creation complete after 2s [id=rtb-053c6b50b33e41890]
aws_route_table_association.app_routes["us-east-1b"]: Creating...
aws_route_table_association.web_routes["us-east-1b"]: Creating...
aws_route_table_association.web_routes["us-east-1a"]: Creating...
aws_route_table_association.app_routes["us-east-1a"]: Creating...
aws_route_table_association.app_routes["us-east-1b"]: Creation complete after 1s [id=rtbassoc-0368ec5fd28dd93a0]
aws_route_table_association.web_routes["us-east-1b"]: Creation complete after 1s [id=rtbassoc-08ca998763a21231e]
aws_route_table_association.app_routes["us-east-1a"]: Creation complete after 1s [id=rtbassoc-0632efaaf242ba6fd]
aws_route_table_association.web_routes["us-east-1a"]: Creation complete after 1s [id=rtbassoc-06f346c929570754e]
vra_cloud_account_aws.lab: Creating...
aws_security_group.app-sg: Creation complete after 4s [id=sg-0e35897dacc752515]
aws_security_group.db-sg: Creation complete after 4s [id=sg-0b293e47f78676b97]
vra_cloud_account_aws.lab: Provisioning with 'local-exec'...
vra_cloud_account_aws.lab (local-exec): Executing: ["/bin/sh" "-c" "sleep 10"]
vra_cloud_account_aws.lab: Still creating... [10s elapsed]
vra_cloud_account_aws.lab: Creation complete after 15s [id=e91c1114-c4ae-4eef-a049-915009a7b3ed]
data.vra_region.lab: Refreshing state...
data.vra_fabric_network.web["us-east-1b"]: Refreshing state...
data.vra_fabric_network.web["us-east-1a"]: Refreshing state...
data.vra_fabric_network.app["us-east-1a"]: Refreshing state...
data.vra_fabric_network.app["us-east-1b"]: Refreshing state...
vra_zone.aws["us-east-1b"]: Creating...
vra_zone.aws["us-east-1a"]: Creating...
vra_image_profile.lab: Creating...
vra_flavor_profile.lab: Creating...
vra_network_profile.app: Creating...
vra_network_profile.web: Creating...
vra_flavor_profile.lab: Creation complete after 1s [id=deec4ef1-d486-488d-adff-f35d1fc749b2-8af76fe6-2161-4658-a1f0-2664d6b917c4]
vra_image_profile.lab: Creation complete after 1s [id=deec4ef1-d486-488d-adff-f35d1fc749b2-8af76fe6-2161-4658-a1f0-2664d6b917c4]
vra_zone.aws["us-east-1b"]: Creation complete after 1s [id=08490e94-e9a3-4efb-b166-d8ef483e2538]
vra_zone.aws["us-east-1a"]: Creation complete after 1s [id=342b16f9-2a58-487e-8494-6b5d6efc76ed]
vra_network_profile.app: Creation complete after 0s [id=bcde0922-564f-41bb-9c63-6dd711213ad3]
vra_network_profile.web: Creation complete after 0s [id=bcd1a144-77a2-4c1d-a70f-0a76db4d34ad]

The network profile still requires a manual tweak to fix the security groups, but the rest of the configurations are good to go. This is a much better process to manage vRA as code and all configurations are tracked in git. I can change a couple of variables and deploy to a new AWS region or utilize from one to all availability zones in that region and have vRA configured to deploy my applications.

Going forward Terraform is going to be the tool of choice for managing my vRA configurations. At the end of a hard day in the lab I can run “terraform destroy” to delete everything to avoid those pesky credit card bills.

The Terraform files used in my experiment are available on github. Please let me know where I’ve done stupid things and how to make things better.

https://github.com/automationramblings/terraform

Network profile needing manual tweak to security groups. Gotta find a fix for this…


Broken SG

Configured Cloud Zones


Cloud Zones

Until next time…

Anytime you learn, you gain. -Bob Ross

 

One thought on “vRA Data Center on Demand with Terraform

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.