{"id":1384,"date":"2020-05-15T13:21:38","date_gmt":"2020-05-15T11:21:38","guid":{"rendered":"https:\/\/blog.besharp.it\/?p=1384"},"modified":"2021-05-21T10:50:14","modified_gmt":"2021-05-21T08:50:14","slug":"python-logging-best-practices-and-how-to-integrate-with-kibana-dashboard-through-aws-kinesis-data-firehose-and-amazon-elasticsearch-service","status":"publish","type":"post","link":"https:\/\/blog.besharp.it\/python-logging-best-practices-and-how-to-integrate-with-kibana-dashboard-through-aws-kinesis-data-firehose-and-amazon-elasticsearch-service\/","title":{"rendered":"Part I: Python logging best practices and how to integrate with Kibana Dashboard through AWS Kinesis Data Firehose and Amazon Elasticsearch Service"},"content":{"rendered":"

Applications running in production<\/strong> lose their ability to tell us directly what is going on under the hood, they become like black boxes and so we need a way to trace and monitor their behavior. Logging<\/strong> is by far the simplest and most straightforward approach to do so. We instruct the program during development to emit information while running that will be useful for future troubleshooting or simply for analysis.<\/span><\/p>\n

When it comes to Python<\/strong>, the built-in module for logging is designed to fit into applications with minimal setup. Whether you\u2019ve already mastered logging in Python or you are starting to develop with it, the aim of this article is to be as comprehensive as possible and to dive into methodologies to leverage logging potentials.\u00a0<\/span><\/p>\n

Note that this is a two-part article. In the second part, we will approach AWS services and use a combination of AWS Kinesis Data Firehose <\/strong>and AWS ElasticSearch Service<\/strong> to provide a way to show and debug logs on Kibana Dashboard<\/strong> as well as storing them to S3 to have long term copies at hand.<\/span><\/p>\n

Python\u2019s logging module basics<\/span><\/h2>\n

Before starting our journey, let\u2019s ask ourselves a simple question: what is a logger?<\/strong> Basically, they are objects a developer interacts with in order to print information. They are the instrument we use to tell the system what to log and how to do it.<\/span><\/p>\n

Given an instance of a generic logger, we can deliver messages wherever we want without worrying about underneath implementation.<\/span><\/p>\n

For example, when we write<\/p>\n

logger.info(\"Logging a variable %s\", variable1)<\/pre>\n
we are describing the following situation:<\/p>\n
\"\"<\/p>\n
Python\u2019s standard library<\/span><\/a> comes with a logging module that you can start using without installing anything.<\/span><\/p>\n
In order to adhere to best practices<\/strong> which request avoiding to simply print into console whatever you want to report, Python\u2019s logging module offers multiple advantages:<\/span><\/p>\n