SHART.CLOUD | DESKTOP | LABROULETTE | GCP-LOG-SINK-BIGQUERY.YAML
01 - Labroulette - Ship Audit Logs to BigQuery Before the Incident
| shart.cloud / labroulette / gcp-log-sink-bigquery |
GCP *OO ~60 min $0-3 reviewed 6/30/2026

Ship Audit Logs to BigQuery Before the Incident

Create a project-level log sink that exports admin activity and data access logs into BigQuery, then query for IAM changes and prove the dataset is locked down to the security team.

#observability#data#iam
Lab AuditSprout, a compliance dashboard that can explain every customer's risk score except who changed IAM last Tuesday, which is inconvenient because last Tuesday is currently on fire. all labs
02 - Actions
score -- - -- votes
-- completed
State loading
03 - Scenario

AuditSprout, a compliance dashboard that can explain every customer's risk score except who changed IAM last Tuesday, which is inconvenient because last Tuesday is currently on fire.

Create a project-level log sink that exports admin activity and data access logs into BigQuery, then query for IAM changes and prove the dataset is locked down to the security team.

Constraints

  • Log sink filter captures IAM changes without dumping every debug log in the project
  • BigQuery dataset is protected from casual project-wide editors
  • Sink writer identity has only the permission needed to write exported logs
  • A sample query finds recent IAM policy changes
Scenario GCP - beginner
05 - Steps
  1. STEP_01

    Create the security dataset

    Create a BigQuery dataset for audit exports with a clear retention and location choice. This is where responders will query facts instead of arguing from screenshots.

    Terraform docs

  2. STEP_02

    Define the filtered log sink

    Create a project sink pointed at the BigQuery dataset with a filter for audit logs that include IAM policy changes. Keep the filter narrow enough that the bill does not become the incident after the incident.

    Hint: Start with Admin Activity logs and IAM method names, then document what your filter includes and what it intentionally leaves out.

    Terraform docs

  3. STEP_03

    Let the sink write, and nobody else

    Grant the sink's generated writer identity BigQuery data editor on the destination dataset. Do not make every project editor a dataset editor just because Terraform is yelling at you.

    Terraform docs

  4. STEP_04

    Grant the security team read access

    Add dataset-level read access for the security group that will investigate IAM changes. Keep operators who deploy apps separate from operators who read the audit archive.

    Terraform docs

  5. STEP_05

    Query an IAM change

    Make a harmless IAM binding change, wait for export, and query BigQuery for the method name, principal email, resource name, and timestamp. That row is your proof the pipeline works.

    Terraform docs

Steps 5 tasks
06 - Deliverables
  • Terraform defining the dataset, logging sink, and dataset-level IAM bindings
  • The exact log sink filter and a short explanation of its scope
  • A BigQuery result showing a recent IAM policy change
Deliverables 3 required
07 - Rubric
Logging sink exports a focused set of audit logs to BigQuery 30%
Sink writer identity has dataset-scoped write access 25%
Security reader access is dataset-scoped and documented 20%
Sample query proves IAM changes are searchable 25%
Rubric self-assessed