Background
Postgres logs and Postgres audit logs are stored in BigAnimal hosted Cloud Storage bucket. This article will introduce how to download the logs.
Prerequisites
When creating the cluster, you can fill in your GCP Project ID and Service Account which will be used for authentication before you can download your logs.
Step 1 Get the Logs information from Monitoring & Logging
Step 2 Get the access key for the service account
Go to your GCP project and generate an access key from the Google Service Account.
Key type is JSON, and key file will be downloaded when clicking CREATE.
Step 3 Download the logs via gcloud CLI
Go to Cloud Shell or log in to gcloud CLI.
$ gcloud config get-value project
$ gcloud auth activate-service-account <Service Account-email address from Step1> \
--key-file=<path to the access key file> \
--project=<GCP project id>
$ gcloud storage ls gs://<OBJECT STORE>/
$ gcloud storage cp gs://<OBJECT STORE>/<pg_cluster_id>/ ./ --recursive
For example
$ gcloud config get-value project
Your active configuration is: [cloudshell-16380]
development-data-38131
$ gcloud auth activate-service-account biganimal-sa-test@development-data-381310.iam.gserviceaccount.com \
--key-file=/home/shanshan_zhao/development-data-381310-b018728185c7.json \
--project=development-data-381310
Activated service account credentials for: [biganimal-sa-test@development-data-381310.iam.gserviceaccount.com]
$ gcloud storage ls gs://pg-logs-tenant-33a90a1c-9ecb-4387-b083-1194b26b86c5
gs://pg-logs-tenant-33a90a1c-9ecb-4387-b083-1194b26b86c5/p-jx5dwd1lnf/
$ gcloud storage cp gs://pg-logs-tenant-33a90a1c-9ecb-4387-b083-1194b26b86c5/p-jx5dwd1lnf/ ./ --recursive
Copying gs://pg-logs-tenant-33a90a1c-9ecb-4387-b083-1194b26b86c5/p-jx5dwd1lnf/2023/09/18/customer_postgresql_cluster.var.log.containers.p-jx5dwd1lnf-1_p-jx5dwd1lnf_postgres-32fc925a352182886c1286c1ec33c14ae4e1c6cc61d9ee64b0ddf9f072d4823f.log/fluent-bit-g8df5-0.json to file://./p-jx5dwd1lnf/2023/09/18/customer_postgresql_cluster.var.log.containers.p-jx5dwd1lnf-1_p-jx5dwd1lnf_postgres-32fc925a352182886c1286c1ec33c14ae4e1c6cc61d9ee64b0ddf9f072d4823f.log/fluent-bit-g8df5-0.json
......
Average throughput: 1.6MiB/s
$ cd p-jx5dwd1lnf/2023/09/18/
$ ls
customer_postgresql_cluster_cst.var.log.containers.p-jx5dwd1lnf-1_p-jx5dwd1lnf_postgres-32fc925a352182886c1286c1ec33c14ae4e1c6cc61d9ee64b0ddf9f072d4823f.log
customer_postgresql_cluster.var.log.containers.p-jx5dwd1lnf-1_p-jx5dwd1lnf_postgres-32fc925a352182886c1286c1ec33c14ae4e1c6cc61d9ee64b0ddf9f072d4823f.log
Possible problem 1
"A Google Cloud Project ID is required for logs and telemetry. Please contact our support team for access to logs and telemetry."
Solution:
Edit Cluster and fill in the GCP project ID and Google Service Account as Prerequisites mentioned.
Possible problem 2
How to upload the access key file to Cloud Shell and how to download the logs files. See the screenshot: