Google's own data: Rakuten and Vodafone Core Web Vitals A/B tests

Google runs a small library of named, dated case studies at web.dev/case-studies, each attached to a real company and a real test. Two of them isolate Core Web Vitals specifically: Rakuten 24, a Japanese e-commerce retailer, and Vodafone Italy. Unlike Deloitte’s 37-site correlation study or Shopify’s aggregate LCP data covered on this site’s page speed and conversion rate piece, these two are single-company A/B tests that Google itself ran or reviewed and published under its own byline. Here is what each one actually tested, in Google’s own words.

Rakuten 24: a month-long split test on one landing page#

The setup. Rakuten’s team picked a landing page with “significant traffic and conversion” so the test could produce a meaningful sample, then split traffic 50/50 for a month: half of visitors saw the original page (version B), half saw a performance-optimized version (version A) that was otherwise unchanged. The write-up, credited to four named Rakuten and Google engineers and last updated 24 August 2022, reports the optimized version’s results relative to the original.

The business numbers. Version A delivered a 53.37% increase in revenue per visitor and a 33.13% increase in conversion rate, alongside a 15.20% increase in average order value, a 35.12% reduction in exit rate, and a 9.99% increase in time on page.

The performance numbers behind it. The same optimized version cut Cumulative Layout Shift by 92.72%, improved First Input Delay by 7.95%, First Contentful Paint by 8.45%, and Time to First Byte by 18.03%, relative to the unoptimized version. Separately, Rakuten also ran a field-data correlation analysis (not the A/B test) across its real user population and found good LCP associated with up to 61.13% higher conversion and 26.09% higher revenue per visitor, a wider, noisier number that supports the same direction without being the headline A/B result.

Vodafone Italy: three specific changes, an 8% sales lift#

The setup. Vodafone’s optimized and original pages ran as a 50/50 A/B split too, each getting roughly 100,000 clicks and 34,000 visits a day from paid display, search, social and app traffic, “visually and functionally identical” apart from the performance changes. Google published this one on 17 March 2021.

What actually changed. Google names three changes, not a general “made it faster” claim: moving the rendering logic for one widget from client-side to server-side, server-side rendering the critical HTML, and hero-image work (resizing, SVG optimization, PNG compression and viewport-based lazy loading for below-the-fold images).

The result. The optimized version improved Largest Contentful Paint by 31% and produced 8% more total sales, plus a 15% uplift in lead-to-visit rate and an 11% uplift in cart-to-visit rate. One honest wrinkle Google’s own page reports: DOMContentLoaded actually increased 15% on the optimized version even as LCP improved, a reminder that one metric moving well does not mean every metric moves the same way.

Why we care#

These are Google’s own selected success stories, not a random sample. Google chose to publish Rakuten 24 and Vodafone Italy specifically because the numbers were good enough to make the point Google wants to make: Core Web Vitals work pays off in revenue, not just rankings. That is a legitimate reason to take Core Web Vitals seriously, and a bad reason to promise a client 53% more revenue from a similar change; Rakuten’s own field data on the same site showed a much wider range of outcomes than its single A/B test headline.

An A/B test beats a before/after comparison, which is why these numbers travel further than Deloitte’s or Shopify’s. Splitting live traffic 50/50 for the same weeks controls for seasonality, marketing pushes and traffic-mix shifts that a simple before-and-after read cannot; that is a stronger methodology than the correlational studies in our companion piece on page speed and conversion rate, even though the two named cases here cover only two companies rather than dozens.

Cite the company and the exact change, not just the percentage. “31% faster, 8% more sales” is Vodafone Italy’s number for three named technical changes on one page in 2021. “53% more revenue per visitor” is Rakuten 24’s number for one landing page over one month. Neither is a general Core Web Vitals multiplier; both are useful, specific, sourced evidence that the work matters.