how.wtf

Set up time-to-live (TTL) attribute in DynamoDB

· Thomas Taylor

How to set up time-to-live ttl attribute in Dynamodb

Amazon DynamoDB’s Time to Live (TTL) feature allows for automatic item deletion after a specified timestamp. It comes at no extra cost and is useful for removing outdated data.

TTL use-cases

  1. Remove stale data and save on DynamoDB memory usage
  2. Retain sensitive data only up to contractual or regulatory obligations
  3. Trigger processes using DynamoDB Streams based on TTL deletions since the deletion stream is marked as “system” rather than normal

How to setup TTL in DynamoDB step-by-step

This tutorial will use Python and the AWS CLI for demonstration purposes. The same logic can be applied to the tooling of your choice.

Create a DynamoDB table with TTL

To keep things simple, we’ll leverage the AWS CLI to quickly create a table.

 1aws dynamodb create-table \
 2    --table-name your-table \
 3    --attribute-definitions \
 4        AttributeName=pk,AttributeType=S \
 5        AttributeName=sk,AttributeType=S \
 6    --key-schema \
 7        AttributeName=pk,KeyType=HASH \
 8        AttributeName=sk,KeyType=RANGE \
 9    --provisioned-throughput \
10        ReadCapacityUnits=5,WriteCapacityUnits=5

then enable TTL

1aws dynamodb update-time-to-live \
2    --table-name your-table \
3    --time-to-live-specification \
4        AttributeName=expires_at,Enabled=true

Output:

1{
2    "TimeToLiveSpecification": {
3        "Enabled": true,
4        "AttributeName": "expires_at"
5    }
6}

If you desire to deploy a table via AWS CDK or CloudFormation, they allow you to specify a TimeToLiveSpecification upon table creation:

 1AWSTemplateFormatVersion: '2010-09-09'
 2Description: CloudFormation template for creating a DynamoDB table with TTL.
 3Resources:
 4  MyDynamoDBTable:
 5    Type: 'AWS::DynamoDB::Table'
 6    Properties:
 7      TableName: your-table
 8      AttributeDefinitions:
 9        - AttributeName: pk
10          AttributeType: S
11        - AttributeName: sk
12          AttributeType: S
13      KeySchema:
14        - AttributeName: pk
15          KeyType: HASH
16        - AttributeName: sk
17          KeyType: RANGE
18      ProvisionedThroughput:
19        ReadCapacityUnits: 5
20        WriteCapacityUnits: 5
21      TimeToLiveSpecification:
22        AttributeName: expires_at 
23        Enabled: true

Add TTL attribute with proper format to an item

The value for a TTL attribute is required to be in Unix Epoch time format. Items with a timestamp older than the current time, except those older by 5 years or more, are marked as expired and will be deleted by the per-partition background scanner.

Here is a Python example demonstrating how to insert an item with correctly formatted TTL:

 1import boto3
 2import time
 3
 4dynamodb = boto3.resource("dynamodb")
 5table = dynamodb.Table("your-table")
 6current_time = int(time.time())
 7
 8# TTL value (e.g., 24 hours from now)
 9ttl_value = current_time + 24 * 60 * 60  # 24 hours from now in seconds
10item = {"pk": "1", "sk": "entity", "expires_at": ttl_value}
11
12response = table.put_item(Item=item)
13print(response)

Output:

 1{
 2  "ResponseMetadata": {
 3    "RequestId": "...",
 4    "HTTPStatusCode": 200,
 5    "HTTPHeaders": {
 6      "server": "Server",
 7      "date": "Fri, 08 Dec 2023 06:24:48 GMT",
 8      "content-type": "application/x-amz-json-1.0",
 9      "content-length": "2",
10      "connection": "keep-alive",
11      "x-amzn-requestid": "...",
12      "x-amz-crc32": "..."
13    },
14    "RetryAttempts": 0
15  }
16}

DynamoDB Query without TTL expired items

Directly from the AWS documentation:

TTL typically deletes expired items within a few days. Depending on the size and activity level of a table, the actual delete operation of an expired item can vary. Because TTL is meant to be a background process, the nature of the capacity used to expire and delete items via TTL is variable (but free of charge).

DynamoDB queries and scans will return the items that are supposed to be expired if they have not been removed from the table. To remediate this, use a filter expression:

 1import boto3
 2import time
 3
 4dynamodb = boto3.resource("dynamodb")
 5table = dynamodb.Table("your-table")
 6current_time = int(time.time())
 7print(current_time)
 8
 9query_params = {
10    "KeyConditionExpression": "pk = :pk",
11    "FilterExpression": "attribute_not_exists(#ttl) OR #ttl > :now",
12    "ExpressionAttributeValues": {":pk": "1", ":now": current_time},
13    "ExpressionAttributeNames": {"#ttl": "expires_at"},
14}
15
16response = table.query(**query_params)
17print(response)

#python   #aws   #dynamodb   #serverless  

Reply to this post by email ↪