This recipe tells you what already happened. The analyzer takes a query you have not run yet and estimates what it will scan, in your browser.
Am I actually using the BigQuery slots I am paying for?
Reservation baseline against the slots your jobs use, hour by hour, and how far a commitment would pay.
Permissions. Needs bigquery.jobs.listAll on the project for JOBS_TIMELINE (BigQuery Resource Viewer has it); bigquery.reservations.list on the reservation administration project (BigQuery Resource Viewer has it); bigquery.jobs.create to run any query (BigQuery Job User has it).
Pick the region your data is in: every INFORMATION_SCHEMA view here is qualified by region.
-- Baseline paid for against the slots this project's jobs used, by hour.
-- RESERVATIONS_TIMELINE has a row for every minute a reservation exists, so
-- idle hours count, as they are billed. It lists the reservations of the
-- project this runs in: run it in the reservation administration project,
-- or put that project's ID in front of the view name.
WITH capacity AS (
SELECT
reservation_id,
TIMESTAMP_TRUNC(period_start, HOUR) AS hour,
AVG(slots_assigned) AS baseline_slots,
SUM(period_autoscale_slot_seconds) / 3600 AS autoscale_slots
FROM `region-us`.INFORMATION_SCHEMA.RESERVATIONS_TIMELINE
WHERE period_start >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
GROUP BY reservation_id, hour
),
usage AS (
-- Script parents repeat their children's slot use; job_creation_time is
-- the partition column, so it is what limits the read.
SELECT
reservation_id,
TIMESTAMP_TRUNC(period_start, HOUR) AS hour,
SUM(period_slot_ms) / 3600000 AS slots_used
FROM `region-us`.INFORMATION_SCHEMA.JOBS_TIMELINE_BY_PROJECT
WHERE period_start >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
AND job_creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 15 DAY)
AND reservation_id IS NOT NULL
AND (statement_type != 'SCRIPT' OR statement_type IS NULL)
GROUP BY reservation_id, hour
),
hourly AS (
SELECT
c.reservation_id, c.hour, c.baseline_slots, c.autoscale_slots,
IFNULL(u.slots_used, 0) AS slots_used
FROM capacity c
LEFT JOIN usage u USING (reservation_id, hour)
)
SELECT
reservation_id,
COUNT(*) AS hours,
ROUND(AVG(baseline_slots), 0) AS baseline_slots,
ROUND(AVG(slots_used), 1) AS mean_slots_used,
ROUND(AVG(autoscale_slots), 1) AS mean_autoscale_slots,
ROUND(COUNTIF(slots_used < baseline_slots) / COUNT(*), 3) AS share_of_hours_below_baseline,
-- Use is at or above these levels in 80% and in 60% of hours.
ROUND(APPROX_QUANTILES(slots_used, 100)[OFFSET(20)], 1) AS slots_used_in_80pct_of_hours,
ROUND(APPROX_QUANTILES(slots_used, 100)[OFFSET(40)], 1) AS slots_used_in_60pct_of_hours,
ROUND(APPROX_QUANTILES(slots_used, 100)[OFFSET(95)], 1) AS p95_slots_used
FROM hourly
GROUP BY reservation_id
ORDER BY mean_slots_used DESC;What it returns
- baseline_slots
- Baseline slots assigned to the reservation, averaged over the hours: billed every hour, used or not.
- mean_slots_used
- Average slots this project's jobs used in an hour, with idle hours counted as zero.
- mean_autoscale_slots
- Average autoscaled slots billed in an hour, from RESERVATIONS_TIMELINE.
- share_of_hours_below_baseline
- Share of hours in which use stayed under the baseline, leaving paid-for slots idle.
- slots_used_in_80pct_of_hours / slots_used_in_60pct_of_hours
- Use is at or above these levels in 80% and in 60% of hours: the most a 1-year and a 3-year commitment can cover and still pay.
- p95_slots_used
- The busy hour. Use above the baseline runs on autoscaled slots, or on idle slots shared by other reservations.
How to read it
- A high share_of_hours_below_baseline, or a mean_slots_used far below baseline_slots, means you are paying for idle capacity.
- Add committed slots only if usage stays above the new level in at least 80% of hours (1-year) or 60% (3-year). At US list prices a committed Enterprise slot costs $0.048 an hour on a 1-year term or $0.036 on 3 years, every hour; the autoscaled slot it replaces costs $0.06, but only in the hours it is used. A p95 above the baseline only means 5% of hours exceed it.
- Use is this project's jobs only. Other projects assigned to the same reservation are not counted, so on a shared reservation the use reads low.
- Utilization of 100% is not the goal. A reservation pinned at its ceiling is queueing work.
Take it further
Run this once, or have it run every day.
Finitizer evaluates this class of question continuously against your live BigQuery estate, tracks how each number moves between runs, and turns findings into assigned tasks. Read-only and keyless.
See how Finitizer works