These four steps describe a relatively basic but very common use case. Now let\u2019s try to draft a list of steps we need to execute in AWS Glue in order to complete the described workflow:<\/p>\n\n\n\n
- Crawl the original database through a JDBC connection.<\/li>
- Use a Glue Job to move the data from the database to S3. Some tables may use bookmarks but others may not.<\/li>
- Crawl the target S3 bucket.<\/li>
- Run a dedicated Glue Spark job to run the join operation on the S3 data lake. Write the results to another S3 partition or bucket.<\/li>
- Crawl the target partition to make the Join results easily queryable with AWS Athena.<\/li>
- Execute the ML Job (SageMaker or the new Glue ML jobs).<\/li>
- Crawl the resulting dataset.<\/li>
- Run a final Glue ETL job to upload the new dataset to the original database.<\/li><\/ol>\n\n\n\n
All these steps need to be executed in the given order, and in case of problems, we would like to be notified and have a simple way to understand what went wrong.<\/p>\n\n\n\n
Without AWS StepFunctions, manually managing these steps would be hellish, and we would probably need an external orchestration tool or to create a Custom orchestration script to be executed on an EC2 or on a Fargate container.<\/p>\n\n\n\n
But why bother? AWS StepFunctions do all this for us, and by being able to directly interact with many AWS services, many integrations are a breeze: for example, with few lines of StepFunctions language, we can catch all the errors in a pipe and forward them to an SNS topic, in order to receive an email in case of error (or a slack notification, SMS or whatever you like more).<\/p>\n\n\n\n
Managing complex flow thus becomes safe and relatively easy. Here is an example of a quite contrived flow:<\/p>\n\n\n\n
![\"AWS](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2021\/04\/image2-625x1024.png\")
Simple StepFunctions flow<\/em><\/figcaption><\/figure><\/div>\n\n\n\n If any of these steps fail we’ll receive an email notification from the SNS topic, have visual feedback of the failed step, and the corresponding logs.<\/p>\n\n\n\n
![\"Example](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2021\/04\/image3.png\")
Example of failed step with logs<\/figcaption><\/figure><\/div>\n\n\n\n StepFunctions thus seems to be a jack of all trades, with a lot of good features, and no significant drawbacks, however, as we all know, this is almost always not true in IT, so which is the catch? <\/p>\n\n\n\n
The real problem for StepFunctions is code management<\/strong>: the step function language is a declarative JSON template (see https:\/\/docs.aws.amazon.com\/step-functions\/latest\/dg\/concepts-amazon-states-language.html<\/a>), which is quite a pain to write and maintain even using dedicated tools such as the visual studio code plugin (see https:\/\/aws.amazon.com\/blogs\/compute\/aws-step-functions-support-in-visual-studio-code<\/a>).<\/p>\n\n\n\n
Furthermore, it would be wonderful to be able to maintain both the StepFunctions code and the Glue Jobs, and the eventual Lambda code in a single integrated project.<\/p>\n\n\n\n
Cloudformation with Troposphere or AWS CDK<\/h2>\n\n\n\n
The most obvious instrument we can use to maintain StepFunctions, the Glue Jobs, and the rest of our ETL infrastructure in an integrated way, is Cloudformation as a deployment tool for everything. However, Cloudformation code is a declarative YML\/JSON language not too different from StepFunctions code, and including that code in these templates is usually quite painful since involves including it as a JSON string in our Cloud Formation YML.<\/p>\n\n\n\n
A much more effective solution is to create our Cloudformation template using a high-level programming language and leveraging the AWS CDK Cloudformation software development framework (https:\/\/aws.amazon.com\/cdk\/<\/a>) which supports many languages (TypeScript, Python, and Java). <\/p>\n\n\n\n
If you decide to use Python, which usually makes sense, since your ETL jobs will probably be written in Python, you also have the option to use Troposphere instead of AWS CDK, as a Cloudformation framework, which is much more versatile in several situations. <\/p>\n\n\n\n
Furthermore, you can author the StepFunctions using the python Step Functions Framework (https:\/\/docs.aws.amazon.com\/step-functions\/latest\/dg\/concepts-python-sdk.html<\/a>) as we\u2019ll do in the following example (Troposphere + Python step function SDK).<\/p>\n\n\n\n
In this very simple example, we want to demonstrate how to create a simple Flow to download Covid Data from a public AWS OpenData S3 bucket, save a small subset of them in a different S3 bucket, and crawl them in order to be prepared for Athena queries. You can extend this basic working example at will! Here is a basic sketch of the infrastructure:<\/p>\n\n\n\n
![\"AWS](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2021\/04\/image5-1024x576.png\")
Basic example infrastructure<\/em><\/figcaption><\/figure><\/div>\n\n\n\n First of all, let\u2019s install the AWS CLI (https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/install-cliv2.html<\/a>) and the python requirements: <\/p>\n\n\n\n
pip install troposphere stepfunctions.<\/pre>\n\n\n\n
Once the installation is done you can download our repository<\/a> and you\u2019ll find a troposphere_main.py<\/strong> file, which contains the troposphere representation of the whole infrastructure (see sketch), other folders containing the python code of the various Lambda functions (start_crawler, check_crawler status), and a README file explaining how to run the project. After this, we\u2019ll need to create an S3 bucket as support for the Cloudformation deployment with the name you prefer.<\/p>\n\n\n\n
Following the instructions in the README, we can just run the main file by running in a console python troposphere_main.py. Executing this script, we\u2019ll compile the python troposphere code to a Cloudformation compliant JSON format. Once this is done, we are ready to run <\/p>\n\n\n\n
aws cloudformation package –template-file troposphere_main.json –s3-bucket <YOUR CLOUDFORMATION S3 BUCKET> –s3-prefix ‘<THE PATH YOU PREFER>’ –output-template-file troposphere_main.yml<\/span><\/p>\n\n\n\n
This command takes as input the JSON file created by Troposphere, and uploads to S3 the Glue and Lambda functions code referenced as local paths giving back another Cloudformation template (this time YML), where the local paths references have been changed in the corresponding S3 references (for further info, take a look at this<\/a>).<\/p>\n\n\n\n
Finally we are ready to deploy the Cloudformation template using the command: <\/p>\n\n\n\n
aws cloudformation deploy –template-file .\/troposphere_main.yml –stack-name testStepfunctionsStack –capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND<\/span><\/p>\n\n\n\n
This we\u2019ll create the testStepfunctionsStack which contains the infrastructure described above. Now you can go to the AWS StepFunctions console and run the newly created function (test-stepfunctions-glue<\/span>), the flow will run its course, and you\u2019ll import the Covid data.<\/p>\n\n\n\n
![\"AWS](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2021\/04\/image4.png\")