The Art and Science of Crafting High-Performing Ads through Experimentation
Crafting high-performing ads doesn’t happen overnight. It’s a meticulous blend of art and science—where data-driven insights meet creative storytelling. That’s powered through hypothesis-driven experimentation. Let’s explore how you can elevate your advertising strategies to not only captivate your audience but also drive substantial revenue growth.
Traditionally, ad creation involves lengthy research, large budgets, and a hope-for-the-best approach, often leading to underwhelming outcomes. There are countless examples, with Solo Brands being a recent example. Where an award-winning campaign failed to generate sales, resulting in the CEO’s firing and their stock price dropping by 50%.
In contrast, I scaled the third-fastest-growing company in the country through rapid, cost-effective experimentation. This modern method uses analytics to pinpoint what captures your audience's attention and drives sales, ensuring every dollar spent is an investment toward measurable success.
A Strategic Blueprint for Ad Success
Measure Key Metrics: Begin by pinpointing the metrics that reflect the ad’s granular performance.
Compare to Historical Benchmarks: Compare these metrics against historical data from your current and past campaigns. This analysis helps identify areas of over and under-performance.
Develop Hypotheses: Based on the available data, craft hypotheses about what changes could enhance ad performance. Whether switching the value proposition, replacing the talent, or refining the problem, each hypothesis will guide your creative iterations.
Craft the Creative Brief: Transform insights and hypotheses into a detailed creative brief. This document should outline every aspect of the ad campaign and what you seek to test.
Review and Refine: Once the brief is set, it’s time to review and refine. The brief ensures all teams are aligned and clear on the campaign’s goals.
Produce the Ads: With a solid brief in hand, move into the production phase. Here, creativity comes to life, shaped by the boundaries of your data-driven strategy.
Run Controlled Experiments: Test your ads in a controlled setting to measure the effectiveness of each variant.
Finally, Repeat
In the era of digital marketing, leveraging advanced analytics is non-negotiable. High-performing ads are the product of both innovative creativity and rigorous analytics. By systematically applying these principles, you can create compelling ads that generate sales and develop your brand. In the fast-evolving landscape of digital advertising, those who leverage both the art of storytelling and the science of analytics are best positioned to thrive.
Step 1 - Measure Key Metrics to Determine Ad Performance
Understanding and refining ad performance is crucial. It’s essential to focus on what really matters: creating ads that captivate your audience and convince them to buy your product. Success in this endeavor requires metrics that tell you why some ads work and others don’t.
Utilizing First-Principles Thinking
Before delving into metrics, it's important to approach ads with First-Principles Thinking. This method involves breaking down complex systems into their foundational elements and reimagining the solution from the ground up. For ads, this means focusing on three key components:
Pitch: Your core message that convinces the audience to buy your product.
Hook: The element that captures the audience’s attention within the first few seconds.
Story: The narrative that memorably illustrates the product's value and defines your brand.
My article, The Winning Formula To Engaging Ads That Generate Millions, breaks down an ad's components into its foundational elements.
Defining and Measuring Key Metrics
Each component of an ad can be analyzed with specific metrics to assess its effectiveness:
Hook Metrics:
3-second view rate: Measures the percentage of viewers watching your ad for over three seconds. A low rate indicates a lack of attention-grabbing in the first few seconds.
Pitch Metrics:
Click-through rate (CTR) Gauges interest by measuring how many viewers click on the ad after seeing it. It indicates whether the pitch is compelling enough to provoke action. However, be warned: CTR can be artificially inflated through clickbait. That’s why we use the aggregate pitch metrics to balance the effect.
Install rate: Specifically for mobile apps, this measures the percentage of visitors to your app store page who proceed to install your app, reflecting how well your ad communicates the app’s value.
Downstream event rate: This tracks how far users engage with your website after clicking the ad. It is useful for non-app advertisements, as it serves the same purpose as the install rate. Choose a conversion funnel event that occurs 20-40% of the time a new user visits your site or app.
Aggregate Pitch Metrics:
Installs per thousand impressions (IPM): The product of CTR times the install rate. This metric helps clarify the overall effectiveness of an ad by balancing the click-through rate with the audience demonstrating genuine interest.
Actions per thousand impressions (APM): The product of CTR times the downstream event rate. This metric helps clarify the overall effectiveness of an ad by balancing the click-through rate with the audience demonstrating genuine interest.
Downstream Pitch Metrics:
First session length: How long users engage with your app in the first 24 hours is the best non-purchase metric I’ve found to indicate initial user satisfaction and likelihood of future value.
Conversion rate: The ultimate measure of an ad’s success, indicating the percentage of impressions that result in a sale.
Story & Pitch Metrics:
Video view rate curve: This curve tracks how long people watch your ad. This metric helps pinpoint where viewers lose interest, allowing you to refine the storytelling aspect. Combine the absolute measures of view rate at 3, 10, and 15 seconds with the percentiles at 25%, 50%, 75%, and completion. You can create the chart by multiplying the percent view rates by the overall length of the ad.
In the below example, I’d review the 5-10-second portion of the video to see what occurred at that point, as it’s likely responsible for the drop in performance.
Engagement metrics (Likes, Shares, & Comments): These indicators measure how viewers enjoyed your ad, offering insights into its entertainment value and emotional resonance.
By meticulously analyzing these metrics, marketers can gain a profound understanding of what aspects of their ads are working and which areas need refinement. This approach streamlines the ad creation process and ensures that each ad works towards achieving higher conversion rates and better overall performance. This scientific approach to creative advertising ensures that your ads are seen and effective in driving measurable business outcomes.
Step 2 - Compare to Historical Benchmarks
In digital marketing, continuously evaluating and benchmarking ad performance against historical data is crucial. This process helps understand which elements of an ad campaign are effective and which need refinement. Let’s delve deeper into how to effectively utilize benchmarks and set control ads during experimentation to optimize ad performance.
Creating Benchmarks
Start by establishing benchmarks based on the performance of your past ads, particularly those that have driven significant spend and engagement. These benchmarks will serve as a standard to measure new ads against and should include key metrics like:
3-second view rate
Click-through rate (CTR)
Install rate or downstream event rate
Installs per thousand impressions (IPM) or Actions per thousand impressions (APM)
Using a Control Ad
For accurate experimentation, use a control—a top-performing ad from a currently or recently active campaign. This ad will help account for variability in performance due to factors such as time of day, audience segment, and how the ad is delivered to the audience. Comparing new ads against this control provides a clear picture of performance deviations.
Comparative Analysis Example
Consider the following example with set benchmarks and results from two different ads:
Advertisement A: This ad shows a lower 3-second view rate but a higher CTR than the benchmark. When combined with the significantly lower install rate, the data suggests the use of clickbait tactics, which attract users but fail to convert. This indicates a need to improve the ad to reflect the product offering.
Advertisement B: Exhibits a strong hook with a higher 3-second view rate than the benchmark but underperforms in CTR. The high install rate suggests that users are highly likely to install once they click, indicating a strong alignment between the ad content and the app’s value proposition. Future ads should refine the pitch to increase CTR while maintaining the quality of the hook and story.
Benchmarking against historical performance and a control ad offers a structured approach to understanding ad effectiveness. Marketers can make informed decisions about modifying their ad strategies for better engagement and conversion by analyzing where ads overperform or underperform relative to benchmarks. This continuous loop of testing, learning, and optimizing is essential for driving success in dynamic digital advertising landscapes.
Step 3 - Develop Hypotheses Behind Ad Performance
After identifying the strengths and weaknesses in an ad's performance through key metrics, the next step is to develop hypotheses on why certain ads underperform or overperform. This involves a deeper analysis using First-Principles Thinking to break down each ad component and evaluate its effectiveness.
First-Principles Thinking in Ad Analysis
First-principles thinking requires us to deconstruct the ad into its foundational elements and question each part's role and effectiveness. This approach ensures a thorough understanding of the ad’s mechanics and interactions with the target audience.
Hypothesis Development Process
Break Down the Ad Components:
Pitch: Evaluate whether the pitch effectively convinces the audience to buy the product. Analyze its clarity, relevance, and persuasive power.
Story: Assess how well the story supports the pitch, entertains the audience, and aligns with the brand identity.
Hook: Examine the initial elements that capture the audience’s attention, such as visuals, sounds, or spokesperson.
Identify Problem Areas: Identify potential problems based on performance data for each component. For example:
If the problem lies in the Pitch, determine if it might be due to targeting the wrong audience, tackling an irrelevant problem, offering a value proposition that fails to resonate, lacking a compelling reason to believe, or failing to communicate what action to take.
For the Story, consider whether it fails to enhance the pitch, fails to be memorable, or doesn’t trigger strong emotion.
With the Hook, check whether the talent fails to connect or the scene fails to build interest.
Formulate Hypotheses: Based on the identified issues, develop specific hypotheses about what might improve performance. Examples include:
Hypothesis for Pitch: "Refining the target persona and altering the problem will improve relevance by targeting a more real issue the persona faces on a daily basis, thereby increasing the click-through rate.” Then, list three alternative personas and problem statements.
Practical Example
Suppose an ad for a fitness app features a pitch that isn't resonating well with the audience, indicated by low engagement and conversion metrics. Here’s how you might develop a hypothesis. Let’s use Solo Brand’s failed Snoop Dogg ad campaign for their smokeless outdoor stove.
Solo Brand’s Original Positioning
Persona: Unknown.
Problem Statement: Smoke bad.
Value Proposition: Smokeless
Reason to Believe: Snoop Dogg is going smokeless.
Solo Brand’s Revamped Positioning
Persona: Individual who loves spending time outdoors with their friends.
Problem Statement: Bonfires are dirty, unhealthy, and make you smell like smoke.
Value Proposition: Have a clean, healthy outdoor adventure with your friends.
Reason to Believe: Some technical reasoning explaining why it works.
Hypothesis: Refined targeting of the pitch to individuals who would benefit from the stove and highlighting immediate user benefits will improve the click-through and conversion rates.
Developing hypotheses based on detailed analysis allows marketers not just to speculate but drive informed experimentation. By understanding the why behind an ad’s performance, you can make direct changes that have the best chance of significantly impacting effectiveness and ROI. This methodical approach to hypothesis generation ensures that each element of your advertising strategy is optimized for maximum engagement and conversion.
Step 4 - Craft the Creative Brief
The creative brief is a foundational tool in advertising that ensures all stakeholders are aligned on the project’s goals, strategies, and expectations. This document facilitates clear communication across diverse teams and serves as a reference point throughout the campaign’s lifecycle. Here’s a guide on how to craft an effective creative brief for your ad campaigns.
Essential Elements of a Creative Brief
General Information
Requestor: Identify who made the request or who is the lead on the project.
Target Channels: Specify where the ad will be displayed (e.g., Meta, TikTok, TV).
Medium: Define the ad format (static, video, user-generated content, carousel).
Estimated Cost: Provide a preliminary budget estimate.
Casting, Location, and Talent: Details about the people and places involved.
Submission and Delivery Dates: Important brief and final deliverables deadlines.
Sign-offs
This section is crucial for accountability. Include dates for when sign-offs are required and who is responsible for each (creative lead, marketing team).
Campaign Goals
Purpose: Clearly state what the campaign intends to achieve.
Expectations: Outline what outcomes are expected from the campaign.
Production Type
Define whether the campaign will involve iterations of previous ads, evolution of existing concepts, or experimentation with completely new ideas.
Hypothesis Overview
Variables Being Tested: Detail what aspects of the hook, pitch, and story are under examination.
Target Metrics: Specify which metrics you aim to improve.
Baseline Rates: Document the current performance levels for these metrics.
Estimated Effects: Predict how the changes might improve these metrics.
Desired Confidence Level: Indicate the statistical confidence level needed (typically 80-95%).
Sample Size and Cost: Estimate the sample size required to achieve the desired confidence level and the associated costs.
Creative Outline
This section should describe the core components of the ad:
Pitch: What is the main message?
Target Persona: Who is the ad aimed at?
Relatable Problems: What issues does your audience face that your product can solve?
Value Propositions: How does your product solve these problems?
Reason to Believe: Why should the audience trust your product?
Call-to-Action: What do you want the audience to do?
Story Elements: Who, what, why, where, and the emotional hooks.
Brand Identity: How the ad reflects the brand’s traits and values.
Hook: How are you going to grab attention?
The Importance of the Hypothesis Overview in a Brief: Do not underestimate the importance of outlining the hypothesis. Let’s consider a scenario where you want to improve the IPM for a mobile app’s advertising campaign.
Baseline IPM: 0.2%
Target Improvement: Increase to 0.22% (a 10% improvement)
Statistical Confidence: 95%
Ad Variants: Three different creative approaches
Cost: $12,000 based on a required sample size of 300,000 impressions at $10 CPM per variant.
Using Bayesian statistics to calculate the required sample size based on the scenario requires 300,000 impressions for each variant. For a total cost of $12,000. The team then aligns on whether a $12,000 cost is worth the potential gain. Completing the math beforehand avoids wasting time and money on experiments with limited impact.
By meticulously outlining every aspect of the campaign, from hypothesis to execution, you prevent miscommunication, align expectations, and set the stage for a successful ad campaign.
Step 5 - Review and Refine the Creative Brief
It’s the final checkpoint before resources are allocated and production begins. This step ensures that only promising advertising concepts are pursued, optimizing both creative impact and budget efficiency.
Evaluating the Briefs During the review, each brief should be evaluated against several key criteria to determine whether it should be approved for production:
Strategic Alignment: Does the brief align with the brand's overall marketing goals and objectives? Ensure it supports broader company initiatives and targets the right audience.
Innovation vs. Iteration: Assess whether the brief proposes genuinely innovative ideas or merely iterative tweaks. Prioritize creative concepts that offer new approaches rather than small modifications to existing ads. This differentiation is crucial as:
Iteration: Small tweaks and changes to an existing concept or idea. These should be limited to less than 5% of resources and only applied to already successful ads to refine further that are early in their life cycle. As the audience gets used to seeing the ad, it loses its effectiveness. New ads must be different from your existing ads to counteract the decay. An iteration will not have the desired effect if done late in its life cycle. Do not waste your time iterating unsuccessful ads. Iterations rarely have a significant impact. I’ve seen companies spin their wheel, never improving, as they focus on color changes and other small tweaks.
Evolution: Allocate 40% to 60% of resources. This involves evolving a successful concept with new twists, talent, stories, or visual elements that refresh the appeal without straying from what made the original successful. These ads are less risky and have a higher probability of success at the trade-off of rarely opening up new audiences or significantly improving performance.
Experimental: Also accounting for 40% to 60% of resources, this category is for entirely new concepts that modify the pitch with respect to a key element. Successful experiments don’t occur often, so expect a high failure rate. But when they do, they deliver large gains.
Resource Allocation: Review the proposed resource allocation to ensure it is proportional to the expected impact. As the guidelines suggest, invest heavily in new concepts that have the potential to outperform existing ones and are supported by a strong hypothesis.
Cost Analysis: Consider the total cost, including production and testing, which should make sense relative to the potential scale and return on investment. High-risk experiments should have the potential for high rewards that justify the gamble.
Hypothesis Strength: The brief must include a well-defined hypothesis about what changes will improve performance and why. This should be based on data-driven insights from previous ad performances and market research.
This stage is crucial for cutting through the clutter of routine iterations and focusing on truly transformative advertising strategies.
Step 6 - Produce new ads
Much of my success in scaling products is due to my creative teams’ ability to produce unbelievably great content. Understanding and optimizing ad production will greatly enhance a company's competitive edge. Here's an overview of the different ad production methods and how to optimize each for maximum efficiency and impact.
1. In-House Production
Graphics Design and Motion Design: This method involves the company’s own team of graphic designers and motion artists producing digital ads. Direct control over the creative process ensures brand consistency and quick turnaround.
Live-Action Video Shoot: This method utilizes the company’s own writers, directors, and production staff to cast, direct, and produce video content. It is highly collaborative and can be tailored to your exact needs. Outsourcing casting and directing to an independent contractor or agency is advisable when internal resources are limited.
Optimization Tips:
Don’t Try to Deliver Everything at Once: Start by choosing the best raw footage from a video shoot. Complete the ads individually or in small batches to minimize turnaround time.
Streamline Communication: Ensure clear, direct communication channels between all creative team stakeholders to minimize misunderstandings and delays.
2. Content Creator or Influencer Collaboration
Working with content creators or influencers allows brands to tap into the creator's audience and creative expertise. This method is governed by a set of loose guidelines the brand provides to ensure the content aligns with the targeted message.
Optimization Tips:
Clear Guidelines: Provide simple briefs that include brand guidelines, campaign goals, and specific deliverables to ensure the content meets expectations. Don’t over-constrain your content creators by limiting their creativity.
Regular Feedback: Implement a structured feedback loop to address any issues early in production.
Pre-Negotiate Revisions: In the initial contract, agree on a maximum number of revisions to the content. It’s cheaper and easier to come to an agreement at the start.
3. External Production (3rd Party Ad Agency or Production Studio)
External agencies bring specialized skills and fresh perspectives to ad production. They handle everything from research and scriptwriting to post-production.
I strongly believe the use of external firms is a mistake. A talented, internal creative team is a competitive advantage. While I understand why companies rely on external agencies and production studios, it’s a missed opportunity to improve their economics versus the competition. Outperforming the competition, even by small margins, delivers enormous economic gains.
My one exception is for television ads. Due to their unique demands, including specific technical requirements and higher budgets, TV ads often necessitate specialized resources that are best managed by external studios or specialized in-house teams. However, that doesn’t mean your team shouldn’t be intimately involved in writing, casting, direction, and production.
General Best Practices for Efficient Production
Turnaround Time: Reducing the time from idea to launch is crucial to tighten the feedback loop. My gold standard is 14 days for in-house statics, 30 days for live-action videos, and 14 days for content creator-led projects.
Resource Allocation: Employ a casting director on retainer to streamline the casting process and ensure a talent pool is always available.
Production Efficiency: When possible, film multiple ad concepts in a single shoot of 6-10 hours to save time and reduce costs. Consider all logistical elements to avoid delays, such as the specific timing for outdoor lighting.
Each ad production method has its strengths and challenges, but the key to success lies in choosing the right approach for the specific campaign needs and optimizing the process to reduce bottlenecks. By strategically managing these elements, a brand can enhance its market responsiveness, significantly accelerate the learning rate, and improve its return on advertising.
Step 7 - Run the experiment
Running experiments to measure the effectiveness of your ads is essential for optimizing performance and ensuring that the most effective ads reach the widest audience. This step goes beyond traditional testing methods to produce a testing approach that is faster and cheaper than traditional A/B testing.
Primer to Ad Experimentation
Only scale your best ads: A 20% improvement for a company with a $20M marketing budget delivers an incremental $4M in value. Small changes in value have huge impacts on performance.
Speed and efficiency are key: Tests should be completed quickly and cheaply.
Expect to fail. A lot: The performance of your ads will mimic a bell curve. You want to scale the outliers, those ads that are roughly two standard deviations better than the average. That means 95+% of the ads you produce will never run at scale.
Phase 1: Setting Up the Experiment
Choose the Channel: Meta (Facebook) is preferred for its ability to find a responsive audience regardless of ad quality and robust analytics.
Select Key Variables: Determine which metrics are most important for your campaign's success. Common choices include:
3-second view rate
Click-through rate (CTR)
Install rate
Downstream event rate
Control Selection: Use a current ad being run at scale as a control to compare against new creative variants. This helps in establishing a baseline for expected performance.
Choose Creative Variants: Choose 3-4 different creatives produced in the previous step.
Set Up the Campaign & Ad Sets: Align the campaign and ad set with your current ads' objectives, targeting, and parameters so they are tested on the same audience.
Phase 2: Run Initial Experiment To Remove Under Performers
Goal: 80-90% of the new creative will underperform the current control. You need to identify the underperforming creatives cheaply and quickly. Thus saving thousands of dollars and days of waiting per creative.
Choose the Metric:
Click-through Rate (CTR)
Installs per thousand impressions (IPM)
Actions per thousand impressions (APM)
Determine Confidence Interval: I suggest using a confidence interval of 90% for initial creative testing. Establish a baseline from the control.
Bayesian Statistics vs. Two-Sided Z-Test: Depending on your team's expertise and resources, you have two statistical methods to choose from. While I began with the traditional two-sided z-test, I later switched to Bayesian statistics.
Automated Analysis: Use tools, such as embedded Python code, to integrate statistical analysis directly into your dashboards, streamlining the process and reducing manual errors.
Example Explained: In an experiment where $300 was spent on each ad, and the control achieved a 1% click-through rate, you are 90% confident that an experimental creative with a CTR of 0.9% is worse than the control or that a CTR of 1.1% is better than the control. Furthermore, doubling the test budget to $600 delivers only a slight increase in confidence at the cost of an additional $300 per creative. I recommend a budget of $250-$400 per creative.
Monitor the budget allocated to each ad variant: Adjustments might be necessary to redistribute budgets if certain ads dominate due to early traction. This helps maintain balanced exposure for all tested creatives. Meta frequently experiences this issue.
Phase 3: Expand To Evergreen
What is Evergreen? Evergreen campaigns are ongoing, scaled campaigns that reach a wide audience.
Review and Decision Making: At the end of the testing phase, evaluate which ads meet the performance thresholds and have the potential to be scaled up. Those that underperform should be phased out.
Promote To Evergreen: Ads that equal to or outperform the controls on the desired metric can be integrated into ongoing evergreen campaigns, ensuring that successful strategies are continually capitalized on. Do not add too many ads to an ad set, as this will result in a suboptimal budget allocation. Further testing continues within evergreen campaigns as they behave differently than test campaigns.
Prioritization of Creative Addition: Decide which creatives to add to the campaign based on their uniqueness, audience reach potential, and performance during testing.
Choose Revaluation Threshold(s): Once running in Evergreen, choose one or more spend levels to reevaluate the performance compared to the other ads running at scale. Use IPM, APM, ROAS, or conversion rate. Make sure to compare the prior 3-7 days' performance, as longer time comparisons can cause bias from higher earlier performance in long-running campaigns. You want to choose the thresholds ahead of time to avoid experimental bias.
Effective ad experimentation is a sophisticated process that requires careful planning, rigorous testing, and analysis. By following a structured approach to experimenting with ad creatives, marketers can significantly enhance the effectiveness of their campaigns, ensuring that only the most impactful ads are scaled up for broader audience engagement.
And… Repeat
The comprehensive approach to crafting, testing, and refining advertisements combines systematic creativity with analytical rigor. That can be continuously repeated to drive the desired results. This iterative process not only enhances the effectiveness of individual ads but also refines the overall marketing strategy. The key takeaway is that a blend of creativity, rigorous testing, and strategic iteration leads to more effective advertising, capable of achieving significant returns on investment and driving brand growth in competitive markets.