Home / Crews Labs
We test everything. Then we test it again.
Crews Labs is how we find out what works before we ask you to rely on it. Here is how we test, what we found, and where the limits are.
The rules are written down before the test is run.
Every result on this page followed the same rules, so a good number means something.
Jobs written first
Tasks, answer keys and scoring rules are saved before any AI sees them, and never edited afterwards.
Websites it had never seen
We judge the browsing on websites nobody tuned for. Real ones, not demos.
Hold-out sites and jobs
We tune on one set, freeze the set-up, then score it on a fresh set held back until the end.
Repeat runs
Most tests run three times, so one lucky answer cannot decide a result.
Scored by code or blind
Code checks answers wherever it can. People who score the rest never see which model made what.
Safe in the real world
On real websites we only look and read. We never sign in or submit anything, and we use made-up customer data.
What we found, in our tests.
Percentages from fixed tests, scored on websites, jobs and photos that nobody had tuned for.
Websites it had never seen
A worker opens a real browser, clicks, filters and reads the page, the way a person does.
What we did
Gave workers real tasks, such as filtering a results page and reading the answer, on websites nobody had tuned for. Most tasks ran three times.
What we found
Reading the whole page beat every shortcut we tried, and keeping the reasoning switched on mattered. Switching it off made it noticeably worse and more costly.
The limit
The websites were UK motor-trade sites, and the tasks were look-and-read only: no sign-ins, nothing submitted.
Look and read only. Our tests never signed in or submitted anything.
From about half, to every job, after tuning
Chains such as find the item, check it, then email the customer, without inventing details or pestering a manager.
What we did
Tried several ways of improving the set-up, then scored the best one on fresh hold-out jobs it had never seen, with the pass rules fixed in advance.
What we found
Extra instructions and more thinking did not help. How the tools were described did. Each model needs its own style, so we apply the change only where it helps.
The limit
The jobs used made-up data. The result belongs to the set-up we tuned and tested.
Percent of fresh multi-step jobs completed correctly. In our tests, on jobs the set-up had never seen.
Cars kept identical
Put a car on a clean backdrop without changing the car: same paint, wheels, condition and number plate.
What we did
Compared several picture tools on real car photos, with results shuffled and scored blind, giving most weight to whether it was still the same car.
What we found
The one we chose kept the car intact in every tested picture and got price text right. Several others invented dents or garbled plates.
The limit
A small set of real photos, judged by eye, one run each. Video has not been tested.
Example from Crews for Motor Trade: same car, new backdrop. Photo by Seamar31, CC0, via Wikimedia Commons.
The same method, applied across the product.
Followed our house rules
Customer replies stayed short, in UK spelling, with one question and no promised price or approval.
In our testsRouted to the right colleague
Incoming messages handed to the right person or worker.
In our testsAttacked as a rival company
Separation between companies was tested by attacking every route as a rival, and reviewed by independent AI security reviewers.
See SecurityCrews separates each company's data. We tested that by attacking every route as a rival company, and it was reviewed by independent AI security reviewers. That is not an audit by an outside firm.
- Attacked as a rival company, route by route.
- Checked again at the database itself.
- Reviewed by independent AI security reviewers.
What these numbers do not tell you.
A number without its limits is marketing. Here are ours.
- iThese are our tests, not a promise. Your websites, systems and jobs may differ, so we measure on yours before we say what a crew will do for you.
- iThe browsing results come from UK motor-trade websites. They were read-and-click tasks. We have not claimed the same figure for every industry or for tasks that need a login.
- iThe test sets are small. Each round used a modest number of tasks, so a result a few points either way is noise, not a difference.
- iThe multi-step jobs used made-up data. The tuning applies to the specific set-up we tested. A different set-up needs its own tuning and its own test.
- iThe picture results are on a small set of real photos. We judged them by eye. Video has not been tested.
- iWebsites change. A result today is not a guarantee next month, which is why we re-test as things change.
- iTesting lowers risk. It cannot prove there is none. That is why important actions still wait for your OK.
- iWe tune how Crews uses a model. We do not claim to have built one.
Ask for the Crews Labs report.
The full report sets out each lab: the tests, how they were scored and what we decided. We will send it, and we are happy to walk through it with you.
Put a crew to work. Keep the final say.
See Crews running a real job for a business like yours, in 30 minutes. Or tell us the job and we will build the crew for you.