AdvancedLesson 26 of 305 min read

Cost and Speed: Planning What a Workflow Really Uses

Estimate monthly cost for three designs with made-up units, see why rules first and saved results save money and read waiting times by median and 95th percentile.

In this lesson you will

  • Estimate cost for different designs
  • Use rules first and saved results sensibly
  • Read median and 95th percentile waits
  • Set budgets and spending alerts

About 25 minutes

The bill nobody planned

Ravi’s assistant works well, and his accountant asks a reasonable question: what will it cost when it handles every email? He has no idea. In testing, a handful of messages cost almost nothing, so the cost never came up. Costs and delays that are invisible at ten messages can become real at ten thousand. Planning them takes ten minutes of arithmetic, and it is the difference between a pleasant surprise and an unpleasant one. This lesson shows the two numbers worth estimating, cost and speed, and three ways to bring them down.

Three levers

  • Rules first. Let cheap, reliable rules settle the easy cases and give only the rest to the costly AI step.
  • Saved results. If the same question arrives again, reuse the saved answer instead of asking again. This is a form of caching.
  • Smaller jobs. Send the AI step only the text it needs, not the whole conversation, because the amount of text sent often decides the price.

Each lever has a catch. Rules must be maintained. Saved answers can go stale when a note or price changes, so they need an expiry. And trimming the text can remove something the step needed. Test after each change, using the saved cases from lesson 24.

Do the sums

The program below uses made-up cost units, where a rule check costs 0.1 per email and an AI step costs 5. It compares three designs for a month of 1,000 emails: everything through the AI step, rules first with the AI step only for the rest, and rules first plus saved results. It also checks each against a made-up budget of 2,000 units, and finally looks at twenty made-up waiting times for the AI step.

import numpy as np

emails = 1000                  # a month, MADE-UP
rule_cost, ai_cost = 0.1, 5    # MADE-UP cost units per email
rule_share = 0.65              # share the rules settle alone
cache_hit = 0.20               # share of AI calls answered from saved results

all_ai = emails * ai_cost
rules_first = emails * rule_cost + emails * (1 - rule_share) * ai_cost
with_cache = emails * rule_cost + emails * (1 - rule_share) * (1 - cache_hit) * ai_cost
print("all by the AI step:", round(all_ai))
print("rules first:", round(rules_first))
print("rules first plus saved results:", round(with_cache))
print(f"saving against all-AI: {1 - with_cache / all_ai:.0%}")
print(f"rules need to settle at least {rule_cost / ai_cost:.0%} to pay off")
budget = 2000
print("fits a 2000 budget (all AI):", "yes" if all_ai <= budget else "no")
print("fits a 2000 budget (rules first):", "yes" if rules_first <= budget else "no")

# MADE-UP waiting times of the AI step, in milliseconds
ms = [820, 790, 905, 850, 1010, 870, 940, 800, 880, 2400, 860, 910, 830, 895, 960, 845, 3100, 875, 920, 810]
print("typical wait (median):", round(float(np.percentile(ms, 50))), "ms")
print("slow wait (95th percentile):", round(float(np.percentile(ms, 95))), "ms")
all by the AI step: 5000
rules first: 1850
rules first plus saved results: 1500
saving against all-AI: 70%
rules need to settle at least 2% to pay off
fits a 2000 budget (all AI): no
fits a 2000 budget (rules first): yes
typical wait (median): 878 ms
slow wait (95th percentile): 2435 ms

Sending everything through the AI step costs 5000 units, which does not fit the budget of 2000 units. Using rules first drops the cost to 1850, which fits, and adding saved results brings it to 1500, a saving of 70% against the all-AI design. The line about the rules shows an interesting fact: they only need to settle 2 per cent of the emails to pay for themselves, because a rule is so much cheaper than an AI call. In practice the question is not whether to use rules, but how much of the work they can safely cover.

Speed is worth a separate look, and the last two lines show why the average is a poor guide. The median, the typical wait, was 878 milliseconds. But the 95th percentile, the wait that one request in twenty exceeds, was 2435 milliseconds, almost three times longer. Two or three slow requests, such as the 2.4 and 3.1 second ones in the list, create that gap. Customers and staff feel the slow ones, so plan for them: set a time limit, as in lesson 23, and decide what happens when it is passed.

A one-page plan for cost and speed

  1. Estimate how many items arrive a month, with a busy month as well as a normal one.
  2. Write the cost of each step, from your tool’s current pricing page, in your own table.
  3. Add up the designs you are considering, and check each against a budget you set in advance.
  4. Set an alert for spending, as lesson 18 described, so a runaway loop shows up the same day.
  5. Recheck every few months, because prices, limits and your volume all change.
Common mistake: planning for a quiet week

Estimating from the first week. Volume grows, and a loop or a retry can multiply the number of calls. Plan for the busy month and set a spending alert.

Made-up sample data

All costs, volumes and waiting times are made up, in invented units. Real prices, limits and speeds depend on the tool, plan and region and change often, so none are quoted. Use your provider’s current information for real estimates.

Choose one workflow. Write down how many items it handles a month, which steps could be replaced by a rule, and the biggest cost you expect. Then set a budget and an alert.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What does “rules first” mean?

  1. Rules replace all testing
  2. The AI step goes first
  3. Rules are optional
  4. Cheap rules handle easy cases and the AI step gets the rest

The AI step is saved for cases the rules cannot settle.

What is caching?

  1. Reusing a saved answer to a repeated question
  2. Deleting old emails
  3. A kind of log
  4. A rule that blocks mail

Saved results avoid asking twice.

What risk comes with saved answers?

  1. They are always wrong
  2. They cost more
  3. They can go stale when notes or prices change
  4. They need no checks

Give them an expiry.

Why look at the 95th percentile of waiting times?

  1. It is the smallest number
  2. It equals the average
  3. It ignores slow requests
  4. A few slow requests hide behind a good median

The typical wait was 878 ms but one in twenty took far longer.

Why plan for a busy month?

  1. Quiet months are free
  2. Volume grows and retries can multiply calls
  3. Budgets never change
  4. Prices fall in busy months

The first week rarely reflects real use.

Summary

  • Cost and speed look small at ten items and large at ten thousand.
  • Rules first and saved results cut cost.
  • The median hides the slow ones, so read the 95th percentile.
  • Set budgets and alerts in advance.

Next lesson

Lesson 27 covers data protection: minimising, mapping and deleting what you hold.

Questions

Where do I find the real cost of an AI step?

On your provider's current pricing page, which changes over time. This lesson uses made-up units to show the method.

Is caching safe?

It can be, with an expiry and a check that the saved answer still fits. Never save private details longer than you need.

What is a percentile?

The 95th percentile is the wait that 95 out of 100 requests stay under. It shows how slow the slow ones are.

Progress is saved in this browser.