To run an AI QA agent on a schedule: pick a rhythm that matches how often your site changes (daily for a busy live site, weekly for one that rarely changes, after every publish if you update often), keep the nightly run short and limited to the journeys your business earns from, and time it far enough before the working day to allow a fix. Alert immediately on a critical bug in a core journey and leave the rest for the morning review. Review the library weekly: update broken cases and archive what no longer matters.
- run website tests automatically every day
- schedule automated website testing
- how do I know my checkout is broken before customers
- nightly test runs what to alert on
- AI QA agent scheduled runs
- daily smoke test for my website
Why run an AI QA agent every night at all?
Thursday evening someone updates a small store add-on. Nobody touches checkout, yet the 'complete order' button stops working on phones. The site opens, the home page is fine, and uptime monitoring shows green. The weekend passes with no orders until a customer calls on Saturday night. A scheduled run exists for this kind of fault: the site is up, but the journey is broken.
Faults also appear without your team changing anything: an outside service changed its behavior, a certificate expired, stock hit zero and broke the product page, an offer's end date passed and a wrong price showed, or a content edit from the admin panel broke a layout. A pre-launch check sees none of it; it all happened afterwards.
- Asks: does the site respond?
- Runs every minute or few minutes
- Catches full outages and slowness
- Cannot tell whether a booking completes
- Asks: can a customer finish what they came for?
- Runs daily, weekly or after a publish
- Catches a broken journey on a working site
- Is not a minute-by-minute monitor
You need both. Uptime is covered in uptime monitoring and SLAs.
How often should a scheduled run happen?
The rhythm follows how often your site changes and what a day of failure costs. Choose:
Pick the one that sounds like you to see the advice.
Every day. One day of a broken purchase journey is a direct loss. Fanann describes this option as a good rhythm for a live app.
Weekly is enough. Pick a day at the start of the working week so someone reads the report and acts.
After every publish. The check runs on the live site right after the change goes out, so you know within minutes whether you broke something.
Weekly, plus an extra run after each update. Time it before work begins on the first working day.
- Rhythm options
- Every day, every week, after every publish
- You set
- The hour, and the weekday for weekly runs
- What runs
- The quick health check or a suite you choose
- Clock
- Riyadh time
- Availability
- Part of higher plans
These are the schedule options in Fanann's QA Engineer. Automatic scheduling is not on every paid plan, so check the pricing page before you build your plan around it.
What should go into the nightly run?
The common mistake is scheduling everything: a long run that eats your allowance and a report nobody reads. The nightly run is a smoke test: short, covering what would cost you money or customers if it broke.
| Run type | What it covers | When it runs |
|---|---|---|
| Quick check | Key pages, buttons and the core journey, changing no data where possible | Every night |
| Full re-check | Every saved case that passed before | Before each release, or weekly |
| New full review | A new plan the agent proposes for a feature or area | When you add a feature, by hand |
| Bug re-test | The cases linked to a fixed bug | After every fix |
To decide what belongs in the nightly check, ask three questions of each case:
- If it failed tonight, would I act before the working day starts? If not, it is not nightly.
- Is it stable? A case that fails one night and passes the next ruins trust in the whole report.
- Does it create data? One order or booking a night is thirty fake records a month.
At what hour should the run happen?
'Every night' need not mean midnight. The right hour meets two conditions: early enough before customers arrive to allow a fix, and after the evening's updates are done.
| Business type | Suggested time | Reason |
|---|---|---|
| Online store | Dawn | Least traffic, and the report is ready with the first coffee |
| Restaurant or delivery | Late morning | The peak is in the evening; the team is awake to fix before lunch |
| Clinic or appointments | Two hours before booking opens | Patients book in the morning |
| Internal system | Before the working day | So staff do not start the day on a broken system |
What deserves an urgent alert, and what can wait until morning?
An alert that wakes you twice for nothing gets muted the third time. Agree on this table in advance:
| What the report shows | What it means | Action |
|---|---|---|
| Critical bug in a core journey | Customers cannot buy or book | Call whoever fixes it, whatever the hour |
| High bug | An important function works wrongly | First task in the morning |
| A bug that 'regressed' | An earlier fix broke | Same day, asking: what changed last? |
| Medium or low bug | Annoying, not blocking | This week's list |
| 'Element not found on the page' | The test needs updating, not the site | Update the case this week |
| 'Test account was rejected' | Setup is incomplete | Update the account details and rerun |
| 'Could not reach the app' | The site did not respond or refused | Check at once: the site may be down |
| Verdict 'inconclusive' | A large share did not run or settle | Do not count it as a pass; find the cause |
For every failed test, Fanann's QA Engineer states whether the cause is a bug in the app, a test that needs review, incomplete setup, or a fault on the platform's own side. That label makes the table workable in two minutes each morning.
How do you keep reports clean and trusted?
A scheduled report loses value if nobody tends it. This is twenty minutes of weekly hygiene:
If an outside developer needs to see a run, plans that include it let you download the report or share a read-only link that expires by itself.
Is it safe to run tests on the live site?
A scheduled run usually targets the live site, which is what customers see. It is safe on these conditions:
- Start with browsing only: the agent opens pages and checks them, skips any test that changes data, and lists it in the report.
- If you allow test data: make it obviously test data (a name such as 'automated test') and agree with operations to ignore and clean it.
- No real payment: write it explicitly under what must never change. Payment is tested in test mode on staging.
- No messages to customers: make sure cases send no email or texts to real numbers.
- A dedicated test account: with normal customer rights, not an admin account.
If you have a staging copy, keep data-changing tests there and the read-only quick check on live. The difference is in staging versus production.
How do you set up the schedule in Fanann?
- 1Build the library first
You cannot schedule what was never tested. Request a mission, review the plan, run it and keep the passing cases. The steps are in the 30-day plan.
- 2Open settings, then 'Test by itself'
Turn on automatic runs for the site or app you want.
- 3Choose the rhythm and time
Every day, every week or after every publish. Then the hour, and the weekday if weekly. The clock is Riyadh time.
- 4Choose what runs
The app's quick health check, or a suite of cases you pick.
- 5Save and watch the first week
Scheduled runs appear in the runs list and in reports. Fix whatever is flaky before you depend on them.
Prepare a quick check that is safe to run every night on the live version of my store, without changing any data. I want to confirm that the home page opens, search returns results, a product page shows the price and the add-to-cart button, the cart shows the total including VAT, and the payment page opens and lists the payment methods. Test at phone size only. Do not register a new account and do not complete any order. Keep the cases few and stable, ordered by their effect on sales.
Try it in FanannWhat routine keeps a schedule useful?
- Every morningTwo minutes
Open the last run's verdict: ready, not ready or inconclusive. If not ready, apply the alert table.
- Every weekTwenty minutes
The report hygiene list: update, archive, and clean out test data.
- Before each releaseA full re-check
Run every saved case by hand; do not wait for the scheduled run.
- Every monthLibrary review
What did the schedule catch before customers? Which complaints did no test catch? Add cases for them.
The best scheduled run is the one you forget because it is green, and believe at once when it turns red.
Key takeaways
- A scheduled run answers one question: does what worked yesterday still work today?
- Keep it short: the quick check daily, the full re-check before a release or weekly.
- Not every red result deserves an alert; separate an app bug from a test that needs updating and from missing setup.
- On the live site: browse only, or clearly labeled test data, and never a real payment.
- Scheduled testing does not replace uptime monitoring, or the reverse; each sees something different.
Frequently asked questions
Can I run my website tests automatically every day?
What is the difference between a scheduled test run and uptime monitoring?
Will daily testing add fake orders to my store?
What do I do when a scheduled test fails but the site works for me?
How many test cases belong in a nightly run?
Do scheduled runs use up my plan allowance?
Ask your AI assistant about this
Copy this prompt into the AI assistant you use, then add the details of your own case.
I am reading this guide: “Running an AI QA agent every night: schedules, alerts and reports” https://fanann.com/en/blog/run-ai-qa-agent-on-schedule/ My question: run website tests automatically every day Use that page as a source and explain the answer step by step for my situation, tell me what to check before I start, and cite the page link.