Implementing a DynamoDB backed API with pagination support solely in API Gateway's Mapping Templates using Terraform

Contrary to what's written on the internet you don't need to run lambda functions in order to implement pagination for DynamoDB in API Gateway. You can easily do it using request mapping templates only. This is a guide to implementing a simple database for news or blog articles with sorting capabilities.

I have created two root level modules:

• aws_dynamodb
• aws_api_gateway

I want to keep costs low so I set up DynamoDB in provisioned mode with r/w capacity of 1. Composite primary key attributes: article type as hash key and timestamp as sort key. I'm using an OpenAPI JSON template version 3 with extensions to easily manage my API in Terraform.

module "aws_dynamodb" {  
  source = "./module/aws/dynamodb"  
  aws_dynamodb_table_hash_key = "article_type"  
  aws_dynamodb_table_name = "article"  
  aws_dynamodb_table_range_key = "created"  
  aws_dynamodb_table_attributes = {  
    article_type : "S",  
    created : "N"  
  }
}

module "aws_api_gateway" {  
  source                                                = "./module/aws/api_gateway"  
  aws_api_gateway_rest_api_name                         = "article"  
  aws_api_gateway_rest_api_path                         = "./template/aws/api_gateway/rest_api/article.tftpl"  
  aws_api_gateway_domain_name_regional_certificate_arn  = module.aws_codebuild["production"].aws_acm_certificate_arn  
  aws_api_gateway_stage_stage_name                      = "articles"  
  aws_api_gateway_base_path_mapping_base_path           = "articles"  
  aws_api_gateway_base_path_mapping_domain_name         = module.aws_api_gateway_domain_name.domain_name  
  aws_api_gateway_rest_api_endpoint_configuration_types = [
    "REGIONAL"
  ]  
}

aws_dynamodb module

module "aws_dynamodb_table" {  
  source         = "./table"  
  billing_mode   = var.aws_dynamodb_table_billing_mode  
  attributes     = var.aws_dynamodb_table_attributes  
  hash_key       = var.aws_dynamodb_table_hash_key  
  name           = var.aws_dynamodb_table_name  
  range_key      = var.aws_dynamodb_table_range_key  
  read_capacity  = var.aws_dynamodb_table_read_capacity  
  write_capacity = var.aws_dynamodb_table_write_capacity  
}

variable "aws_dynamodb_table_attributes" {  
  description = <<EOT  
    (Required) Set of nested attribute definitions. Only required for hash_key and range_key attributes. See below.  
  EOT  
  type        = map(string)  
}  

variable "aws_dynamodb_table_billing_mode" {  
  description = <<EOT  
    (Optional) Controls how you are charged for read and write throughput and how you manage capacity.  
    The valid values are PROVISIONED and PAY_PER_REQUEST. Defaults to PROVISIONED.  
  EOT  
  default = "PROVISIONED"
}  

variable "aws_dynamodb_table_name" {  
  description = "(Required) Unique within a region name of the table."  
  type        = string  
}  

variable "aws_dynamodb_table_hash_key" {  
  description = <<EOT  
    (Required, Forces new resource) Attribute to use as the hash (partition) key.  
    Must also be defined as an attribute. See below.  
  EOT  
  type        = string  
}  

variable "aws_dynamodb_table_range_key" {  
  description = <<EOT  
    (Optional, Forces new resource) Attribute to use as the range (sort) key.  
    Must also be defined as an attribute, see below.  
  EOT  
  type        = string  
}  

variable "aws_dynamodb_table_read_capacity" {  
  description = <<EOT  
    (Optional) Number of read units for this table.  
    If the billing_mode is PROVISIONED, this field is required.  
  EOT  
  default = 1
}  

variable "aws_dynamodb_table_write_capacity" {  
  description = <<EOT  
    (Optional) Number of write units for this table.  
    If the billing_mode is PROVISIONED, this field is required.  
  EOT  
  default = 1
}

aws_dynamodb_table module

resource "aws_dynamodb_table" "aws_dynamodb_table" {  
  billing_mode   = var.billing_mode  
  name           = var.name  
  hash_key       = var.hash_key  
  range_key      = var.range_key  
  read_capacity  = var.read_capacity  
  write_capacity = var.write_capacity  

  dynamic "attribute" {  
    for_each = var.attributes  
    content {  
      name = attribute.key  
      type = attribute.value  
    }  
  }  
}

variable "attributes" {  
  type = map(string)  
  description = <<EOT  
    (Required) Set of nested attribute definitions. Only required for hash_key and range_key attributes. See below.  
  EOT  
}  
  
variable "billing_mode" {  
  description = <<EOT  
    (Optional) Controls how you are charged for read and write throughput and how you manage capacity.  
    The valid values are PROVISIONED and PAY_PER_REQUEST. Defaults to PROVISIONED.  
  EOT  
  type = string  
}  
  
variable "name" {  
  description = "(Required) Unique within a region name of the table."  
  type = string  
}  
  
variable "hash_key" {  
  description = <<EOT  
    (Required, Forces new resource) Attribute to use as the hash (partition) key.  
    Must also be defined as an attribute. See below.  
  EOT  
  type = string  
}  
  
variable "range_key" {  
  description = <<EOT  
    (Optional, Forces new resource) Attribute to use as the range (sort) key.  
    Must also be defined as an attribute, see below.  
  EOT  
  type = string  
}  
  
variable "read_capacity" {  
  description = <<EOT  
    (Optional) Number of read units for this table.  
    If the billing_mode is PROVISIONED, this field is required.  
  EOT  
  type = number  
}  
  
variable "write_capacity" {  
  description = <<EOT  
    (Optional) Number of write units for this table.  
    If the billing_mode is PROVISIONED, this field is required.  
  EOT  
  type = number  
}

aws_api_gateway module

I have left out the implementation of aws_api_gateway_deployment, aws_api_gateway_stage, aws_api_gateway_base_path_mapping, and aws_api_gateway_domain_name for brevity. If you need help implementing these modules you can visit the documentation page or reach out to me for help.

module "aws_api_gateway_rest_api" {  
  source                       = "./rest_api"  
  name                         = var.aws_api_gateway_rest_api_name  
  endpoint_configuration_types = var.aws_api_gateway_rest_api_endpoint_configuration_types  
  body                         = templatefile(var.aws_api_gateway_rest_api_path, {})  
}  
  
variable "aws_api_gateway_rest_api_name" {  
  description = <<EOT  
    (Required) Name of the REST API. If importing an OpenAPI specification via the body argument,  
    this corresponds to the info.title field. If the argument value is different than the OpenAPI value,  
    the argument value will override the OpenAPI value.  
  EOT  
  type = string  
}  
  
variable "aws_api_gateway_rest_api_endpoint_configuration_types" {  
  description = <<EOT  
    (Required) List of endpoint types. This resource currently only supports managing a single value.  
    Valid values: EDGE, REGIONAL or PRIVATE. If unspecified, defaults to EDGE.  
    Must be declared as REGIONAL in non-Commercial partitions.  
    If set to PRIVATE recommend to set put_rest_api_mode = merge to not cause the endpoints and  
    associated Route53 records to be deleted. Refer to the documentation  
    for more information on the difference between edge-optimized and regional APIs.  
  EOT  
  type = list(string)  
}  

variable "aws_api_gateway_rest_api_path" {  
  description = "path to the openapi spec template"  
  type = string  
}  

aws_api_gateway_rest_api module

resource "aws_api_gateway_rest_api" "aws_api_gateway_rest_api" {  
  name = var.name  
  body = var.body  
  
  endpoint_configuration {  
    types = var.endpoint_configuration_types  
  }  
}

variable "body" {  
  description = <<EOT  
    (Optional) OpenAPI specification that defines the set of routes and integrations to create as part of the REST API.  
    This configuration, and any updates to it, will replace all REST API configuration  
    except values overridden in this resource configuration and  
    other resource updates applied after this resource but before any aws_api_gateway_deployment creation.  
    More information about REST API OpenAPI support can be found in the API Gateway Developer Guide.  
  EOT  
  type = string  
}  
  
variable "name" {  
  description = <<EOT  
    (Required) Name of the REST API. If importing an OpenAPI specification via the body argument,  
    this corresponds to the info.title field. If the argument value is different than the OpenAPI value,  
    the argument value will override the OpenAPI value.  
  EOT  
  type = string  
}  
  
variable "endpoint_configuration_types" {  
  description = <<EOT  
    (Required) List of endpoint types. This resource currently only supports managing a single value.  
    Valid values: EDGE, REGIONAL or PRIVATE. If unspecified, defaults to EDGE.  
    Must be declared as REGIONAL in non-Commercial partitions.  
    If set to PRIVATE recommend to set put_rest_api_mode = merge to not cause the endpoints and  
    associated Route53 records to be deleted. Refer to the documentation  
    for more information on the difference between edge-optimized and regional APIs.  
  EOT  
  type = list(string)  
}

output "id" {  
  value = aws_api_gateway_rest_api.aws_api_gateway_rest_api.id  
  description = "ID of the REST API"  
}  
  
output "root_resource_id" {  
  value = aws_api_gateway_rest_api.aws_api_gateway_rest_api.root_resource_id  
  description = "Resource ID of the REST API's root"  
}  
  
output "body" {  
  value = aws_api_gateway_rest_api.aws_api_gateway_rest_api.body  
  description = <<EOT  
    (Optional) OpenAPI specification that defines the set of routes and integrations to create as part of the REST API.  
    This configuration, and any updates to it, will replace all REST API configuration  
    except values overridden in this resource configuration and  
    other resource updates applied after this resource but before any aws_api_gateway_deployment creation.  
    More information about REST API OpenAPI support can be found in the API Gateway Developer Guide.  
  EOT  
}

Endpoint configuration

I have created three endpoints for retrieval of articles:

• /type/{type}
  • Query all of the articles in the hash key and order in reverse chronological order. Each page is limited to 10 articles.
• /type/{type}/start/{start}
  • Use the end of the previous page as the start of the current one and again limit the page to 10 articles. Present it in reverse chronological order.
• /type/{type}/created/{created}
  • Retrieve an article using composite primary key of hash key and sort key.

I'm using projection and hiding the output of the body of the documents for the first two endpoints to keep the payload size small.

GET /type/{type} integration

Set up your endpoint to use DynamoDB service. Fill in credentials, HTTP method is POST, action Query.

Mapping template

{  
  "TableName": "article",  
  "KeyConditionExpression": "article_type = :type",  
  "ExpressionAttributeValues": {  
    ":type": {  
      "S": "$util.escapeJavaScript($input.params('type'))"  
    }  
  },  
  "ScanIndexForward": false,  
  "Limit": 10,  
  "ProjectionExpression": "created, title, thumbnail, description, view_count"  
}

GET /type/{type}/start/{start} integration

Same set up as in the previous endpoint configuration. We're using the ExclusiveStartKey field to input our previous position.

Mapping template

{  
  "TableName": "article",  
  "KeyConditionExpression": "article_type = :type",  
  "ExpressionAttributeValues": {  
    ":type": {  
      "S": "$util.escapeJavaScript($input.params('type'))"  
    }  
  },  
  "ScanIndexForward": false,  
  "Limit": 3,  
  "ExclusiveStartKey": {  
    "created":{  
      "N":"$util.escapeJavaScript($input.params('start'))"  
    },  
    "article_type":{  
      "S":"$util.escapeJavaScript($input.params('type'))"  
    }  
  },  
  "ProjectionExpression": "created, title, thumbnail, description, view_count"  
}

GET /type/{type}/created/{created} integration

Mapping template

{  
  "TableName": "article",  
  "KeyConditionExpression": "created = :created and article_type = :type",  
  "ExpressionAttributeValues": {  
    ":created": {  
      "N": "$input.params('created')"  
    },  
    ":type": {  
      "S": "$util.escapeJavaScript($input.params('type'))"  
    }  
  }  
}

Test

You can open a file from Google Drive, Dropbox or GitHub by opening the Synchronize sub-menu and clicking Open from. Once opened in the workspace, any modification in the file will be automatically synced.