![]() ![]() ![]() Create a directory structure inside the S3 bucket to store the historical data from the partitioned and standalone tables.Create partitioned and standalone (non-partitioned) tables in the RDS for PostgreSQL instance.To build out this solution, this blog post walks you through the steps listed below: This solution helps set up the pipeline to join historical data with current data in the system using Amazon Athena. The diagram also shows querying the current data from Amazon RDS for PostgreSQL using AWS Lambda. However, you can use other formats as well.Ī user queries the historical data from Amazon S3 via the AWS Glue Data Catalog, using Amazon S3 as the data source. In the following diagram, the client is an Amazon Elastic Compute Cloud (Amazon EC2) instance where you can install your application, or install the utilities like psql client, AWS Command Line Interface (AWS CLI), and others.įor this solution, we’ll upload historical data to Amazon S3 in the form of CSV files. In this post, we walk through how to move archived data from Amazon Relational Database Service (Amazon RDS) for PostgreSQL to Amazon Simple Storage Service (Amazon S3), fetch historical (archived) data from S3, and use SQL’s to write queries that join data between Amazon RDS for PostgreSQL and Amazon Athena. Developers need a solution that lets them benefit from using cheaper storage for archived data, but they can still use SQL queries to join data between the active and archival systems. However, there are often business requirements where an application must query both active data and archived data simultaneously. While databases are used to store and retrieve data, there are situations where applications should archive or purge the data to reduce storage costs or improve performance. ![]()
0 Comments
Leave a Reply. |