{"id":5761,"date":"2023-04-14T09:00:00","date_gmt":"2023-04-14T07:00:00","guid":{"rendered":"https:\/\/blog.besharp.it\/?p=5761"},"modified":"2023-04-13T16:29:58","modified_gmt":"2023-04-13T14:29:58","slug":"efficiently-stream-dml-events-on-aws","status":"publish","type":"post","link":"https:\/\/blog.besharp.it\/efficiently-stream-dml-events-on-aws\/","title":{"rendered":"Efficiently stream DML events on AWS"},"content":{"rendered":"\n
In the past, many workloads came with their own big and monolithic database, where not only the application but also reporting tools and technical support connected to it and performed queries. <\/p>\n\n\n\n
As this is still true today, companies are moving towards having single information stored on multiple data sources and servers. Only the core application should be able to access the database, reporting tools should use data that is stored on a separate instance, and monitoring and data analytics should be done by aggregating data that comes from different sources.<\/p>\n\n\n\n
To do this we need to stream the changes that are occurring on our database to one or more destinations. Today we are going to take a look at how to do this on AWS.<\/p>\n\n\n\n
AWS Database Migration Service (DMS)<\/strong> is a powerful tool for migrating data between various database platforms. One of the standout features of AWS DMS is its Change Data Capture (CDC) functionality, which allows for real-time streaming of changes made to a source database to a target database.<\/p>\n\n\n\n When using AWS DMS, you have the option to attach a target database directly as an endpoint or use Amazon Kinesis Data Streams to capture and process the streaming data. <\/p>\n\n\n\n Here are some differences between the two approaches:<\/p>\n\n\n\n Overall, both approaches have their advantages and disadvantages, and the best choice depends on your specific use case and requirements. In this article, we are going to explore the possibility to process insert\/update\/delete events on flight with the help of Amazon Kinesis Data Streams.<\/p>\n\n\n\n Now, let\u2019s build a proof of concept to test out the CDC streaming solution with DMS and Kinesis Data Streams. The idea is to have an automated process that gives us an easy way to replicate changes that happen on a source database to one or more destination engines.<\/p>\n\n\n\n This is a diagram of what we\u2019re going to build:<\/p>\n\n\n\n <\/p>\n\n\n\n
Setup efficient DML events streaming on AWS <\/h2>\n\n\n\n