The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs, or forwards them for further analysis. It was originally developed in C due to its low footprint and high performance. Recently Python language bindings were added, so you can collect, process and store log messages using embedded Python code.

Python bindings always have two parts: a configuration part – where you configure a source, parser, or destination – and the actual Python code part. You can store the code in-line in the syslog-ng configuration file, or in external files, if you have a larger project.

The first Python binding added to syslog-ng was the Python destination. Many databases and cloud services have an easy to use Python API, while a C implementation of the API is either missing or difficult to implement. The syslog-ng application handles log messages and parsed data as name-value pairs. You can pass name-values to the Python destination code in mulitple ways: an object which contains all of the name-value pairs, or as a dict, which contains only name-value pairs configured in syslog-ng.conf. Recent releases also support the use of templates.

Next the Python parser was implemented. Originally it was intended as an easy-to-code parser for messages not supported by the built-in parsers, like PatternDB or csv-parser. But you can also use it to enrich log messages from external databases. For example, you can use it to resolve host names from IP addresses.

Recently support for Python source was added. You can use it to fetch data or implement a server in Python, for example, to collect messages over HTTP or Kafka protocols.

Obviously the Python code is slower than native C code. On the other hand, extending syslog-ng in Python offers a lot more flexibility than development in C. You do not need a development environment other than a text editor and the Python module installed. Neither do you need you to compile the code. Just write it and use it.

It is easy to get started, as there is no need for extra setup. You can get proof-of-concept minimal setup working with a single method quickly. You can extend it later to have proper initialization and connection control for added robustness.

Learn how to customize your log processing infrastructure in Python using syslog-ng, integrate your logs to local and cloud services, and the implementation details and benefits of such solutions.