Let Your Agent Ship a Landing Change, Then Read It 7 Days Later

A landing change often gets shipped and never read. A week later nobody remembers the exact date, the traffic mix has moved, and the page stays by default. This workflow fixes that by deciding the rule before the change goes out and putting the date where your agent can find it.

You need a site with tracker.js installed, an Agent Analytics project, and an agent that can edit your repo. Reading results works through any agent connected to Agent Analytics, including Grok Bot over MCP.

The numbers in the examples are from a demo recipe app and are made up.

Step 1: pick one change and one goal

Start from a diagnosis, not a hunch:

Use Agent Analytics to diagnose where my-site loses users before activation. Start from project context, use the configured activation events, run a funnel and session paths, break down the largest leak, then recommend one narrow change. Name the goal event it should move.

Keep it to one thing: one headline, one call to action, one pricing line. Use a real outcome as the goal, such as signup, not page_view. If the goal event is a button click rather than a completed signup, ask your agent to confirm that the completed signup is tracked separately.

Step 2: write the decision rule before you ship

This is the step people skip. Ask your agent to write the rule down and save it next to the change:

Before we change anything, write a keep-or-revert rule for this landing change and save it in the commit message and in docs/experiments/hero-headline.md. Use the goal event I named. Compare the 7 days after shipping with the 7 days before, same weekdays, leaving out the day of the change. Include a minimum sample, a keep threshold, a revert threshold, and what makes the result unreliable.

A rule you can start from. The thresholds are examples, so set them for your own traffic:

Goal: signup, counted per visitor
Windows: 7 days before vs 7 days after, ship day excluded

Minimum sample: at least 200 visitors in each window and at least
20 signups across both. If not met, extend by 7 days, once.

Keep:    signup rate up 15% or more (relative)
Revert:  signup rate down 10% or more (relative)
Between: no evidence of a gain, so revert

Unreliable, extend instead of deciding:
- another annotated change landed in either window
- one traffic source went from under 10% to over 30% of visitors

Two details matter. The “between” case defaults to revert, so a change needs evidence to stay. And the rule names what would make you distrust the result, which stops you explaining away a bad number afterward.

Step 3: ship it, with your review

Make exactly this change on a branch: [describe the one change]. Do not touch anything else. Show me the diff and wait for my OK before you push or deploy.

Read the diff yourself. You are approving a change to your live site.

Step 4: tag it in project context

Agent Analytics project context can hold dated annotations for major changes such as a landing page, pricing or an experiment. They come back with analytics results for that period, so a later read knows a change happened. Ask your agent to read the existing context first and keep what is already there:

Read the project context for my-site, then add one annotation without removing existing entries: occurred_at is the time the change went live, title is "Hero headline changed", and the note says "One change: headline copy only. Rule saved in docs/experiments/hero-headline.md. Read on day 7." Show me the merged context before saving it.

Annotation titles can be up to 120 characters and notes up to 500. Keep them short, and do not store secrets or personal data. See CLI for the format.

Step 5: read it on day 7

Seven days after the change went live, ask:

Read the hero-headline change on my-site. Use the annotation date and the rule in docs/experiments/hero-headline.md. Compare the 7 days after with the 7 days before, same weekdays, ship day excluded. Show visitors, goal events and the rate for each window, check the minimum sample, check the source mix and any other annotated changes in either window, then give one answer: keep, revert, or extend. State the sample sizes and say that a before and after comparison cannot prove the change caused the move.

Come back on day 7 and paste that prompt. Scheduling lives in the agent you use, not in Agent Analytics. Grok Bot can run a saved prompt on a recurring schedule, as described in the Grok Bot guide, for example a weekday traffic summary. The docs describe repeating routines, not a one-time day-7 job, so treat the day-7 read as a prompt you run when the window closes.

Here is the shape of a good answer, from a demo recipe app with made-up numbers:

Hero headline change, shipped Nov 20
                 Before (Nov 13-19)   After (Nov 21-27)
Visitors                 1,000               1,100
Signups                     30                  44
Signup rate               3.0%                4.0%   (+33% relative)

Minimum sample: met (1,000 and 1,100 visitors, 74 signups)
Source mix: google.com 40% before, 42% after. No source over 30% that was not before.
Other annotated changes in either window: none
Rule result: keep (up 15% or more)
Caveat: before and after, not a controlled test.

And one that should not be called:

Visitors 150 before, 160 after. Signups 6 and 8.
Minimum sample: not met (under 200 visitors per window, 14 signups).
Result: extend 7 days. Rate moved from 4.0% to 5.0%, but 2 signups is within noise.

Step 6: act on it

If the answer is keep, record it:

Add an annotation to my-site project context, merged with the existing entries: "Hero headline kept after 7-day read". Then delete docs/experiments/hero-headline.md or mark it closed.

If the answer is revert, do it with the same review as the ship:

Revert the hero headline change on a branch, show me the diff, and wait for my OK before you push. Then add an annotation to project context, merged with the existing entries: "Hero headline reverted after 7-day read".

When you need a controlled test

A before and after read shares one weakness: anything else that changed that week is mixed in. If you are on a Pro plan and have enough traffic, run the change as an A/B experiment instead. Visitors are split between the old and new version at the same time, so seasonality and traffic mix affect both sides equally. The flow is the same: pick one goal, create the experiment, QA both variants, then read results.

Create an experiment for the hero headline on my-site with control and new_headline variants, using signup as the goal event. Show me how to force each variant so I can QA both before any traffic sees them.

Check results for the hero headline experiment and tell me whether we have enough data to pick a winner. If we do, recommend whether to keep running it, pause it, or complete it with a winner.

Creating an experiment is a write action, so you review and approve it before it runs, and shipping the variant on your site is a separate step. The full walkthrough is in the experiment tracking guide.

Docs

• AI Agent Experiment Tracking: goal events, declarative variants, forced-variant QA, results.
• Closed-loop growth analysis: the diagnosis prompt and the readout prompt.
• CLI: project context and annotations.
• Ask Grok Bot how your site is doing: connecting Grok Bot and recurring scheduled prompts.