This recipe tells you what already happened. The analyzer takes a query you have not run yet and estimates what it will scan, in your browser.
Which queries are scanning partitioned tables without a partition filter?
Find the queries that read every partition of a partitioned table.
That is usually the single largest avoidable cost in a BigQuery estate.
Permissions. Needs bigquery.jobs.listAll on the project for JOBS (BigQuery Resource Viewer has it); bigquery.tables.get and bigquery.tables.list for COLUMNS, TABLE_STORAGE (BigQuery Metadata Viewer has them); bigquery.jobs.create to run any query (BigQuery Job User has it).
Pick the region your data is in: every INFORMATION_SCHEMA view here is qualified by region.
-- Queries that touched a partitioned table but billed a suspiciously
-- large fraction of it. Heuristic: bytes billed within 20% of the table's
-- total logical size means little or no pruning happened.
-- COLUMNS is region-qualified like the other views, so every dataset in
-- the region is checked, not one.
WITH partitioned AS (
SELECT DISTINCT table_schema, table_name
FROM `region-us`.INFORMATION_SCHEMA.COLUMNS
WHERE is_partitioning_column = 'YES'
),
sizes AS (
SELECT table_schema, table_name, total_logical_bytes
FROM `region-us`.INFORMATION_SCHEMA.TABLE_STORAGE
),
jobs AS (
SELECT
job_id, user_email, creation_time, total_bytes_billed,
ref.dataset_id AS table_schema,
ref.table_id AS table_name,
SUBSTR(REGEXP_REPLACE(query, r'\s+', ' '), 0, 300) AS query_preview
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT,
UNNEST(referenced_tables) AS ref
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
AND job_type = 'QUERY'
AND IFNULL(statement_type, '') != 'SCRIPT'
AND state = 'DONE'
AND error_result IS NULL
)
SELECT
j.job_id, j.user_email, j.table_schema, j.table_name,
j.total_bytes_billed,
s.total_logical_bytes,
ROUND(SAFE_DIVIDE(j.total_bytes_billed, s.total_logical_bytes), 3) AS fraction_of_table,
j.query_preview
FROM jobs j
JOIN partitioned p USING (table_schema, table_name)
JOIN sizes s USING (table_schema, table_name)
WHERE SAFE_DIVIDE(j.total_bytes_billed, s.total_logical_bytes) > 0.8
ORDER BY j.total_bytes_billed DESC
LIMIT 50;What it returns
- fraction_of_table
- Bytes billed ÷ the table's total logical size. Near 1.0 means a full scan.
- total_logical_bytes
- The whole table, uncompressed, the denominator.
How to read it
- Finds full scans that read most columns. A full scan of a few columns bills a small share of the table and does not show here.
- The ratio is a heuristic, not proof. A query selecting every column of every partition scores ~1.0; so does one selecting one column across a table where that column is most of the bytes. Read the query preview before filing a ticket.
- The fix is almost never "add a WHERE clause" in isolation; it is to set require_partition_filter on the table so the next person cannot make the same mistake.
- total_bytes_billed is for the whole job, not for this table, so a job that also reads other large tables can score high, even above 1.0, while this table was pruned. Check the other tables in the query before blaming this one.
Take it further
Run this once, or have it run every day.
Finitizer evaluates this class of question continuously against your live BigQuery estate, tracks how each number moves between runs, and turns findings into assigned tasks. Read-only and keyless.
See how Finitizer works