Module 15 — Creative Testing and Performance Learning

Category: Professional Delivery | Duration: ~60 min | Prerequisites: Module 13

Learning Objective

By the end of this module, you will be able to design creative A/B tests that change one variable at a time, distinguish creative hypotheses from verified performance findings, use platform-provided metrics responsibly, document what was tested and observed, and write recommendations supported by available data or clearly labeled as hypotheses.

1. Principle: Test One Variable at a Time

Rule: Change one major creative variable at a time when practical.

Testing multiple variables simultaneously makes it impossible to know which change drove the result. This isn't about perfection — it's about learning.

Hypothesis vs. Verified Finding

Never invent analytics. If you don't have data, label it as a hypothesis. If you have data, report it accurately.

What You Can Test

What You CanNOT Test (Ethically)

• Misleading thumbnails to trick clicks
• False claims about book performance
• Invented reader statistics
• Artificially inflated engagement (comment pods, etc.)

2. Using Platform Metrics Responsibly

TikTok Metrics

YouTube Shorts Metrics

Instagram Reels Metrics

Interpreting Retention Curves

A healthy retention curve for book-promotion:

100% ┐
     | \
     |  \
  80% |   \___          Good: sharp hook, gradual drop, slight rise at CTA
     |       \
     |        \
  60% |         \___
     |              \
     |               \
  40% |                \___
     |                    \
     |                     \
  20% |                      \____
     |                           \
     └─────────────────────────────────
     0s   5s   10s  15s  20s  25s  30s

Red flags:

• Sharp drop in first 1 second → hook failed
• Gradual decline with no recovery → no payoff
• Sharp drop at CTA → CTA is too abrupt or unclear

3. Visual Demonstration

Professional Example — A/B Test Plan

Book: Avoid Construction Disputes Objective: Discovery (find your next read)

Variable tested: Hook structure

Observed:

• Variant A: 65% retention at 3s, 32% at 15s
• Variant B: 78% retention at 3s, 41% at 15s

Conclusion: Surprising premise outperformed question-based hook for this audience. Next experiment: Test opening visual — aerial site vs. contract close-up.

Weak Example — Invented Data

A/B Test Report:
- Test: Different thumbnails
- Result: Variant B got 50,000 more views
- Conclusion: Thumbnails are the most important factor

Why it fails: No sample size given, no statistical significance, no context about other changes, no platform specified. The conclusion is overstated and potentially misleading.

3b. Common Mistakes and Corrections

4. Step-by-Step Workflow

1. Choose a Variable

1. Review past performance — which videos performed well?
2. Identify a single variable to test
3. Define the variants — A (control) and B (change)
4. Ensure both variants are otherwise identical

2. Write the Test Plan

1. State the hypothesis clearly
2. Define the expected outcome
3. List the success metric (retention, engagement, etc.)
4. Note the sample size needed for significance
5. Set the test duration

3. Execute and Measure

1. Post both variants (same time, same audience if possible)
2. Wait for minimum impressions (1,000+ recommended)
3. Record the observed results
4. Note any confounding factors

4. Document and Decide

1. Compare the metrics
2. State whether the hypothesis was confirmed
3. Document the next experiment
4. Note any limitations of the test

5. Quality Gate — Test Report

• [ ] One variable changed
• [ ] Both variants otherwise identical
• [ ] Hypothesis stated before execution
• [ ] Metrics accurately reported
• [ ] No invented data or significance
• [ ] Next experiment documented
• [ ] Limitations acknowledged

5. Creative Testing Categories

Hook Testing

Test different hook structures for the same book:

1. Question hook — "What do you see inside an ambulance?"
2. Surprising premise hook — "911 calls don't look like TV."
3. Relatable problem hook — "You've seen the ambulance speed past."

Opening Visual Testing

1. Recognition-first — start with the book cover
2. Story-first — start with a scene from the concept
3. Problem-first — start with the reader's pain point

Story Structure Testing

1. Linear — hook → context → escalation → payoff → reveal → CTA
2. Loop — ends feed back into the opening
3. Question-and-answer — poses a question, answers it, leads to CTA

CTA Testing

1. Soft discovery — "Discover the book."
2. Medium traffic — "Explore the full description."
3. Hard conversion — "Check the current offer."

Use one CTA type per variant.

6. Hands-On Exercise

Prepare two genuinely different creative variants for the book assets/books/crossing-the-line.md and write a short testing plan.

Variant A

• Hypothesis: Starting with the passport tension will outperform starting with the cheese anecdote.
• Hook: Question-based — "Why do they sometimes send you in and sometimes not?"
• Opening visual: Passport in trembling fingers
• Story structure: Linear

Variant B

• Hypothesis: Starting with the relatable problem will outperform the tension hook.
• Hook: Relatable problem — "You've packed that wheel of cheese. Did you declare it?"
• Opening visual: Cheese wheel on inspection counter
• Story structure: Linear

Testing Plan

Complete the Testing Plan Template with:

1. Variable being tested (what's different between A and B)
2. Expected result and reasoning
3. Success metrics (which retention/CTR numbers to compare)
4. Sample size requirement
5. Test duration
6. How you'll document results
7. Label as hypothesis or data-driven

7. Assessment

Scored assessment (25 points total):

Pass threshold: 22/25

Submission Requirement: Submit the two creative variants, the testing plan, and a clear hypothesis for each. Recommendations must be supported by available data or clearly labeled as hypotheses.

Downloadable Resources

• Testing Plan Template
• Production Report Template