Deploying Flowable infrastructure on AWS

This document describes deploying a Flowable kubernetes cluster and it's required components on AWS using Terraform.

The example deployment configuration files can be downloaded here.

Terraform

Terraform is an open source Infrastructure-As-Code (IAC) tool. Such tools allow you to manage infrastructure with configuration files rather than through a graphical user interface. More information regarding Terraform can be found here.

In order to provision the required Flowable infrastructure a collection of cloud platform specific Terraform modules are provided; including for AWS. These modules can be executed with a certain configuration that can be tailored to specific needs. For example; sizing, database type, permissions etc.

A complete example containing a configuration of a development environment and a production / live environment is provided. Deploying this example is described below.

Architecture Overview

AWS account organization

Security Account

When managing multiple environments (dev, qa, prod) in the cloud there are several options on how to organize this. This guide follows the approach of having separate accounts for each environment and a centralized security account. This account will contain;

1. The Terraform state
2. The Identity and Access Management (IAM) users and user groups

Terraform state

Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. By default, Terraform stores state locally in a file. When working with Terraform in a team, use of a local file makes Terraform usage complicated because each user must make sure they always have the latest state data before running Terraform and make sure that nobody else runs Terraform at the same time. With remote state, Terraform writes the state data to a remote data store (S3 bucket), which can then be shared between all members of a team.

IAM users and user groups

The IAM Console enables you to create child users separate from your root account. These can either be service users with API access, or full-on user accounts (with access to the AWS Management Console) that you can use for employee accounts.

The target environments on which the Flowable infrastructure will be deployed will be separate accounts that will have IAM roles defined. These roles have attached policies defining the permissions of that role. When Terraform wants to perform an action on an environment it will authenticate using the credentials (access key) of a IAM user defined in the central security and will assume the role of the target environment. Security policies attached to user groups define which user is allowed to assume what role on which environment.

This approach has the benefit of having one central location of managing access to the different environments and storage of the Terraform state.

Environment Accounts

Each Flowable environment (dev, qa, prod) will be under a separate AWS account. This ensures separation of resources.

Prerequisites

Before we can start deploying the infrastructure will need to complete some one-time initial steps. To perform these steps you need to have access to the AWS management console and have sufficient rights to create (root) accounts.

Creating the AWS accounts

For this example we will need 3 additional accounts;

• security account
• dev environment account
• prod environment account

Use the AWS Organisation service in the AWS management console (in the main account; containing your user) to add these 3 accounts. For each account a valid email address needs to be provided. Provide OrganizationAccountAccessRole as the IAM role name. This ensures that a role with admin privileges within this account exists that can be assumed. This role can, by default, only be assumed by users from the main account.

When using the AWS management console to access resources in different accounts it can be useful to make use of switch role links. Read the documentation provided here on how to create and use them.

Setup CLI tooling

The initial account setup scripts make use of the AWS Command Line Interface (AWS CLI) tooling. In addition Terraform and Terragrunt need to be installed. Terragrunt is a thin Terraform wrapper that provides some additional features like templating. More information on Terragrunt can be found here.

Homebrew

On MacOS systems the package manager Homebrew can be used.

Install Homebrew.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

For more information about Homebrew and how to install can be found here.

Terragrunt and Terraform

brew install terragrunt

this will also install Terraform. Please make sure Terraform is installed when Terragrunt is installed using another method.

More information about Terragrunt and how to install can be found here. More information about Terraform and how to install can be found here.

AWS CLI

brew install awscli

More information about AWS CLI and how to install can be found here.

After installing the AWS CLI can be configure with your security credentials. For this you will need the Access Key ID and the AWS Secret Access Key belonging to your user (from the main account).

For example;

aws configure --profile flw-iac

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-west-1
Default output format [None]: json

More information on how to configure AWS CLI can be found here.

In addition to configuring the AWS CLI credentials we need to configure some named profiles in order to execute AWS CLI commands on the different target accounts.

Edit the file ~/.aws/config in your favorite editor. Add the following section to add a named profile for each account.

[profile flw-sec]
role_arn = arn:aws:iam::<security_account_id>:role/OrganizationAccountAccessRole
source_profile = flw-iac
region=<region>

[profile flw-dev]
role_arn = arn:aws:iam::<development_account_id>:role/OrganizationAccountAccessRole
source_profile = flw-iac
region=<region>

[profile flw-prod]
role_arn = arn:aws:iam::<production_account_id>:role/OrganizationAccountAccessRole
source_profile = flw-iac
region=<region>

The source_profile refers to an entry in the ~/.aws/credentials file. When you already had AWS CLI configured (or when a different profile name was used) look up the profile entry containing the needed credentials an refer to that profile here.

More information on how to configure named profiles can be found here.

Initial accounts setup

Bootstrapping the accounts with the required setup for deploying the Flowable infrasture using Terraform can be done with scripts included in the flowable-iac-deployments project. The scripts are located in the aws/init/ folder. For each account; security, development and production there is a folder containing a Make file that will execute the CloudFormation templates. These templates contain the definition of the required initial resources.

Initial setup scripts

Update configuration

Before you can execute the bootstrapping scripts some configuration values specific to your environment needs to be updated in the main Make file.

Open aws/init/Makefile located in the flowable-iac-deployments project with your favorite editor.

## BEGIN CONFIGURATION
##
## SECURITY ACCOUNT
SEC_ACCOUNT_ID := <security account ID>
AWS_CLI_SEC_PROFILE := <AWS CLI profile name security account; f.e flw-sec>

FLOWABLE_REPO_AUTH_USERNAME := <Flowable Docker / Helm repository username>
FLOWABLE_REPO_AUTH_PASSWORD := <Flowable Docker / Helm repository password>
FLOWABLE_DEV_LICENSE_FILE := <Flowable development license file location>
FLOWABLE_PROD_LICENSE_FILE := <Flowable production license file location>

## DEVELOPMENT ACCOUNT
AWS_CLI_DEV_PROFILE := <AWS CLI profile name development account; f.e flw-dev>
DEV_ACCOUNT_ID := <development account ID>

## PRODUCTION ACCOUNT 
AWS_CLI_PROD_PROFILE := <AWS CLI profile name production account; f.e flw-prod>
PROD_ACCOUNT_ID := <production account ID>

## END CONFIGURATION
...
...
...

Update the value between < > to match your environment.

Run scripts

In the folder aws/init/Makefile execute the following commands to bootstraps the environments;

1. Provision Development account with the required IAM roles and IAM policies

make init-development

2. Provision Production account with the required IAM roles and IAM policies

make init-production

3. Provision Securition account with the required IAM roles and IAM policies, the Terraform backend resources and the Flowable secure resources.

make init-security

Environment Architecture

Both quick start Flowable infrastructures (prod and dev) consist of the following AWS services and resources.

• Virtual Private Cloud network (VPC). This logically isolates all Flowable resources from other virtual networks in AWS Cloud. It is organized with private and public subnets with associated security groups and routing tables.

• Elastic Kubernetes Service (EKS). This service provides a managed Kubernetes control plane and a autoscaling nodegroup.

• Relational Database Service (RDS). This service provides a managed relational database. The configured database instance type differs per environment.

• OpenSearch Service. This service is the successor to Amazon Elasticsearch Service and provides a managed Elasticseach 7.10 cluster. This service is optional and not configured for the Development environment by default.

Development

Production

Deploy Environment

Quick start projects structure

The Flowable Quick start can be downloaded here.

The archive consists of 2 folders;

• flowable-iac-deployments
• flowable-iac-modules

Flowable IAC Deployments

This folder contains the Terragrunt configuration files that, along with the Terraform modules in flowable-iac-modules, can be used to deploy a complete infrastructure running Flowable on various cloud platforms.

The code in this repo uses the following folder hierarchy:

account
└ region
  └ environment
    └ layer
      └ resource

Where:

• Account: For each provider account. In this case non-prod and prod.

• Region: Within each account, there will be one or more AWS regions, such as us-east-1, eu-west-1, and ap-southeast-2, where you've deployed resources.

• Environment: Within each region, there will be one or more "environments", such as dev, stage, prod, etc. Typically, an environment will correspond to a single AWS Virtual Private Cloud (VPC), which isolates that environment from everything else in that AWS account.

• Layer: This is a logical set of resources. F.e. foundation which consist of the base networking, security and Kubernetes cluster resources.

• Resource: This are the actual AWS services and resources. F.e. VPC, RDS or Flowable App.

Flowable IAC Modules

This folder contains a selection of Terraform modules that, along with the Terragrunt configuration files in flowable-iac-deployments can be used to deploy a complete infrastructure running Flowable on various cloud platforms. Some static configuration is present in these module definitions but most will be provided by the environment specific configuration defined in flowable-iac-deployments.

Update deployment configuration

During the initial bootstrapping of the AWS accounts some resources were created. These resources have unique Amazon Resource Names (ARNs). These ARNs need to be updated in some parts of the configuration.

Update the following properties;

property	description	filename
account_id	Update with the Development account ID	/flowable-iac-deployments/non-prod/account.hcl
sec_account_id	Update with the Security account ID	/flowable-iac-deployments/non-prod/account.hcl
aws_profile	Update with the AWS CLI profile; f.e. flowable-iac	/flowable-iac-deployments/non-prod/account.hcl
tf_assume_role_arn	Replace the account ID with the Development account ID	/flowable-iac-deployments/non-prod/foundation/foundation.hcl
tf_assume_role_arn	Replace the account ID with the Development account ID	/flowable-iac-deployments/non-prod/app/app.hcl
account_id	Update with the Production account ID	/flowable-iac-deployments/prod/account.hcl
sec_account_id	Update with the Security account ID	/flowable-iac-deployments/prod/account.hcl
aws_profile	Update with the AWS CLI profile; f.e. flowable-iac	/flowable-iac-deployments/prod/account.hcl
tf_assume_role_arn	Replace the account ID with the Production account ID	/flowable-iac-deployments/prod/foundation/foundation.hcl
tf_assume_role_arn	Replace the account ID with the Production account ID	/flowable-iac-deployments/prod/app/app.hcl

Deploying the infrastructure

There are several options for deploying the infrastructure depending on 'where' (in what folder) you execute the terragrunt commands.

You can deploy multiple resources / layers / environments using the terragrunt run-all apply command depending. Or you can deploy a single resource using the command terragrunt apply.

Another option is to preview te change that will be deployed comparing the current configuration against the live infrastructure using the command terragrunt plan. Mind: this will not work for modules that have dependencies to other modules that have not been 'applied' yet. More info on this can be found here.

The next section shows some examples on how to deploy or undeploy complete environments or parts of the infrastructure.

Deploy (complete) development and production environment

This will deploy the complete development environment.

cd flowable-iac-deployments
terragrunt run-all apply

Deploy (complete) development environment

This will deploy the complete development environment.

cd flowable-iac-deployments/aws/non-prod/eu-central-1/dev
terragrunt run-all apply

Undeploy (complete) production environment

This will undeploy the complete development environment.

cd flowable-iac-deployments/aws/prod/eu-central-1/dev
terragrunt run-all destroy

Deploy foundation layer development environment

This will deploy the Virtual Private Cloud (VPC), the Elastic File Service (EFS), the Elastic Kubernetes Service (EKS) with the required components. It will also create a Resource Group grouping all deployed services and resources.

cd flowable-iac-deployments/aws/non-prod/eu-central-1/dev/foundation
terragrunt run-all apply

Review (plan) and deploy (execute) Flowable development environment

This will deploy the Flowable Helm chart using a development environment specific configuration defined in a custom values.yaml file.

cd flowable-iac-deployments/aws/prod/eu-central-1/dev/app/flowable-app
terragrunt plan

Have a look at the output and review the planned changes to the live infrastructure.

cd flowable-iac-deployments/aws/prod/eu-central-1/dev/app/flowable-app
terragrunt apply

Add IAM user(s)

When the initial bootstrapping is completed we can add IAM users that will be able to interact with deployed resources like the Kubernetes cluster.

Per environment there are 3 user groups that allow users within these groups to assume roles on the environments. These roles provide different permissions to the users that assume these roles.

The following groups were create during the initial account bootstrapping.

1. DevOps admin group; allows users to assume a role providing all required privileges to deploy and manage a complete Flowable infrastructure.
2. Flowable admin group; allows users to assume a role providing admin privileges within the 'Flowable namespace' within the EKS Kubernetes cluster.
3. Flowable user group; allows users to assume a role providing read only privileges within the 'Flowable namespace' within the EKS Kubernetes cluster.

The users can be added using the AWS Management Console. We will add a 'role switch' link to the console so that we can assume the OrganizationAccountAccessRole role in the Security account. This provides us with the needed permissions to add IAM users.

For this example we will add a user that will be part of the prod and dev DevOpsAdmin groups.

1. Login using the credentials of your main account
2. Add a 'role switch link' using this url; (replace the account id)

https://signin.aws.amazon.com/switchrole?roleName=OrganizationAccountAccessRole&account=<security account id>

3. Provide a name for the 'role switch link'. For example; Security Admin

4. After confirming the role will get assumed. The current assumed role will be visible in the upper right corner;

5. Navigate to the IAM service.

6. Select Users under Access management and click Add users.

7. Provide a username; for example; flowable_devops_user. For AWS credential type select Access key - Programmatic access. Proceed to next step.

8. Select the groups dev_dev_ops_admin and prod_dev_ops_admin. Proceed to the review step.

9. Verify the choises made and select create user.

10. IMPORTANT: The secret access key presented here will only be provided once. Do not close the window before completing the next steps.

11. Add the credentials to the AWS CLI configuration. Open ~/.aws/credentials with your favorite editor and add the following section;

[flowable-devops-admin]
aws_access_key_id = <use generated value>
aws_secret_access_key = <use generated value>
region = <use the current region; for example; eu-central-1>

12. Add the assume role profile to the AWS CLI configuration. Open ~/.aws/config with your favorite editor and add the following section;

[profile flowable-devops-admin-dev]
role_arn = arn:aws:iam::<use dev account id>:role/flowable/DevOpsAdminRole
source_profile = flowable-devops-admin

[profile flowable-devops-admin-prod]
role_arn = arn:aws:iam::<use prod account id>:role/flowable/DevOpsAdminRole
source_profile = flowable-devops-admin

AWS SDK dependency for managing IAM service

The AWS SDK for Java provides an API for AWS services. One of those services is the Identity & Access Management (IAM) service. Therefore, managing IAM Users requires the AWS SDK dependency.

The AWS SDK dependency can be populated in Flowable Helm in the values.yaml file (example below). This would fetch the mvn artifact and put it on the classpath of the container, after that it can be used and configured as if it was included in the image.

...
   jdbc:    
     fetchDriver: true
     mvnDependency:  
       groupId: com.amazonaws
       artifactId: aws-java-sdk
       version: 1.12.468
... 

(replace the version with the correct version)

Advanced scenarios

The following section describes some more advanced use cases with a more in depth explanation.

Change Flowable datasource

The quick start environments dev and prod deploy and configure different RDS databases. In the dev example PostgreSQL databases are configured. The prod example configures Oracle databases. The actual databases are deployed by the Terragrunt (which delegate to Terraform) modules that are located here;

flowable-iac-deployments/aws/non-prod/eu-central-1/dev/app/rds-postgres/

and

flowable-iac-deployments/aws/prod/eu-central-1/prod/app/rds-oracle/

Each Flowable component has a dedicated database instance. Which instances are created for the prod example is managed by the properties defined in;

flowable-iac-deployments/aws/prod/eu-central-1/prod/app/app.hcl

locals {
  ...
  
  # enable Flowable components
  # this will also affect DB instance creation / deletion
  flowable_control_enabled = true
  flowable_design_enabled  = false
  flowable_work_enabled    = true
  flowable_engage_enabled  = false

  ...
}

The deployment of the Flowable application components is handled by the Flowable Helm chart. The configuration of the database source is done by overriding the default Helm chart configuration in the values.yaml file located here;

flowable-iac-deployments/aws/prod/eu-central-1/prod/app/flowable-app/values.yaml

work:
  enabled: ${flowable_work_enabled}

...

  jdbc: 
    fetchDriver: true
    mvnDependency:
      groupId: com.oracle.database.jdbc
      artifactId: ojdbc8
      version: 19.3.0.0
  
...

  envVariables:
    spring.datasource.url: ${spring_datasource_url_work}
    spring.datasource.hikari.minimumIdle: 10 
    spring.datasource.hikari.maximumPoolSize: 100

...

In addition to the datasource configuration the download of the JDBC driver jar is configured here. With fetchDriver set to true the defined Maven dependency is fetch with a init container before the pod initialization. The jar is made available to the Flowable component by adding it to the Java classpath inside the container.

The values.yaml is a template file that will be processed by Terragrunt. The spring.datasource.url is set with a variable ${spring_datasource_url_work}. This value is defined in the Terragrunt configuration here;

flowable-iac-deployments/aws/prod/eu-central-1/prod/app/flowable-app/values.yaml

# Include the root terragrunt `root.hcl` configuration. The root configuration contains settings that are common across all
# components and environments, such as how to configure remote state.
include {
  path = find_in_parent_folders("root.hcl")
}

...

dependency "database" {
  config_path = "../rds-oracle"
}

...

inputs = {
  cluster_name                     = dependency.k8s.outputs.cluster_name
  release_name                     = local.workload
  release_namespace                = local.namespace
  chart_name                       = "flowable"
  chart_version                    = "3.11.3"

  values = templatefile("values.yaml", {
    ingress_domain                      = "${local.env_suffix}.${local.external_domain}"
    spring_elasticsearch_rest_uris      = (local.es_enabled ? "http://${dependency.elasticsearch.outputs.elk_endpoint}" : ""),
    flowable_indexing_index-name-prefix = "${local.workload}-",
    es_enabled                          = local.es_enabled
    flowable_control_enabled            = local.flowable_control_enabled
    flowable_design_enabled             = local.flowable_design_enabled
    flowable_work_enabled               = local.flowable_work_enabled
    flowable_engage_enabled             = local.flowable_engage_enabled
    spring_datasource_url_control       = (local.flowable_control_enabled ? "jdbc:oracle:thin:@${dependency.database.outputs.db_control_endpoint}/${dependency.database.outputs.db_control_name}" : ""),
    spring_datasource_url_design        = (local.flowable_design_enabled ? "jdbc:oracle:thin:@${dependency.database.outputs.db_design_endpoint}/${dependency.database.outputs.db_design_name}" : ""),
    spring_datasource_url_work          = (local.flowable_work_enabled ? "jdbc:oracle:thin:@${dependency.database.outputs.db_work_endpoint}/${dependency.database.outputs.db_work_name}" : ""),
    spring_datasource_url_engage        = (local.flowable_engage_enabled ? "jdbc:oracle:thin:@${dependency.database.outputs.db_engage_endpoint}/${dependency.database.outputs.db_engage_name}" : "")
  })
}
  

The value of the spring_datasource_url_work variable is an output of the RDS Oracle database module dependency.