Skip to main content

How to use pgBadger to analyze BigAnimal Postgres Logs

kun liu
  • Updated

pgBadger is a popular PostgreSQL log analyzer. This article describes how to use pgBadger to analyze BigAnimal logs stored in Azure Blob storage and AWS S3 bucket.

Prerequisites

  1. BigAnimal has several options for logging solutions. Before exporting logs, make sure you are using the custom logging solution which store Postgres logs in Azure Blob storage or AWS S3 bucket base on the cloud provider you use.
  2. In order to get meaningful database analysis reports, you must enable and set up log configurations in your BigAnimal cluster.

Install pgBadger

Clone the pgBadger git repository and install

git clone https://github.com/darold/pgbadger.git
cd pgbadger/
perl Makefile.PL
make && sudo make install


Export BigAnimal logs from Azure Blob Storage

  1. Get Azure Blog Storage name from BigAnimal Portal,  Clusters - <Cluster Name> - Monitoring & Logging page:

    Azure_Blob.png

  2. Log into the Azure portal and go to Storage accounts, find the storage by the name you got from the BigAnimal portal:

    Azure_Blob_name.png

  3. Click BigAnimal storage, click Containers in the left menu, and BigAnimal cluster logs are stored in a container named biganimal-logs.

    Azure_Blob_log.png
  4. Inside the container, logs can be found based on the BigAnimal cluster ID contained in the log file name.

    Azure_BA_log.png

Export BigAnimal logs from AWS S3 Bucket

  1. Get AWS S3 bucket name from BigAnimal Portal, Clusters - <Cluster Name> - Monitoring & Logging page:

    AWS_S3.png

  2. Log in to AWS management console, go to Amazon S3 service, and locate the bucket through the bucket name obtained from the BigAnimal portal:

    AWS_S3_bucket.png

  3. Click the kubernetes-logs object to find the cluster log by the cluster ID. If your cluster is highly available, you will see log objects for each pod.

    AWS_S3_log.png

Generate pgBadger report

  1. Run pgBadger by setting the format log type to jsonlog with option -f and output file with option -o. If you want to parse multiple logs, you can pass the log directory as a value to pgBadger. 
    pgbadger -f jsonlog <log_files_directory> -o <output.html>
  2. A HTML file will be generated as the pgBadger report:

    pgBadger.png