Digital Shelf Analytics: Fueling CPG Ecommerce Growth with Public Online Data
The Ecommerce Imperative for CPG Brands
The meteoric rise of ecommerce has dramatically reshaped the retail landscape in recent years. Global online sales topped $4.2 trillion in 2020 and are expected to grow to $6.5 trillion by 2023, according to eMarketer. The COVID-19 pandemic accelerated this trend, with ecommerce‘s share of total retail sales jumping from 13.6% in 2019 to 18% in 2020.
For consumer packaged goods (CPG) brands, this rapid shift to online purchasing has made ecommerce a front-and-center strategic priority. Pre-pandemic, ecommerce accounted for only 4.4% of total CPG sales in the U.S. By the end of 2020, that share had surged to 11.4% – a staggering 5-year leap in just 9 months.
As more consumers move online for even routine purchases, CPG brands have had to quickly adapt to an ecommerce-first reality. Winning in this new environment means ensuring your products are easily found, attractively presented, competitively priced, and always available wherever your customers are shopping.
But for large brands selling across dozens or even hundreds of retailer websites and marketplaces, keeping track of your online presence and performance is a monumental challenge. Teams can easily spend hours scouring different sites each day to check product descriptions, search rankings, reviews, inventory levels, and competitor activity. This manual, small-sample approach simply doesn‘t scale in a world of endless online shelf space.
This is where digital shelf analytics comes in. By automatically collecting and analyzing public online data at scale, DSA gives brands real-time visibility and control over their entire ecommerce presence. Let‘s take a closer look at how it works.
Under the Hood of Digital Shelf Analytics
At its core, digital shelf analytics (DSA) is about turning the vast amounts of public data spread across ecommerce sites into actionable insights brands can use to optimize performance. DSA platforms achieve this through a sophisticated web scraping and data processing pipeline:
Data collection: The DSA platform deploys bots or "web crawlers" to visit specified retailer websites and marketplaces at regular intervals – typically daily, but sometimes as often as hourly. Using rotating proxy networks and headless browsers to avoid detection, these crawlers systematically navigate to product and category pages, search result pages, and more. They then extract and save the HTML underlying each page.
Parsing and data structuring: Raw HTML by itself isn‘t very useful. DSA platforms use advanced parsing techniques to identify and extract relevant data points from the code, like product titles, descriptions, prices, search rankings, reviews, and stock levels. Natural language processing (NLP) helps classify unstructured text. All of this data is then organized into structured formats like JSON that can be stored in databases and acted on programmatically.
Data integration: To unlock the full value of ecommerce data, DSA platforms typically integrate it with data from a brand‘s internal systems like ERPs, PIMs, and inventory management tools. This allows performance and operations to be analyzed holistically in a single interface.
Analysis and insight generation: With all this structured data in a centralized repository, DSA platforms apply business intelligence and data visualization tools to surface insights and recommendations. This can include:
- Dashboards comparing a brand‘s Share of Search and Share of Shelf across retailer sites
- Alerts for product pages with missing or inconsistent content
- Identification of trending search terms and fast-rising competitor products
- Analysis of ratings and reviews to gauge customer sentiment
- Forecasting models to predict demand and avoid stockouts
Through this process of large-scale, automated public data collection and AI-powered analysis, DSA arms brands with near real-time intelligence they can use to take control of their digital shelf presence, make better decisions, and grow online sales.
Driving Online Growth with DSA: Key Use Cases
DSA insights can inform ecommerce strategy and execution at every stage of the customer journey. Let‘s explore some of the most impactful applications:
Search optimization: 68% of online shoppers use site search to find products. DSA reveals exactly which search terms your products are showing up for across retailer sites, as well as their organic and sponsored rank. Armed with this intel, brands can tweak product titles, descriptions, and keywords to show up in more relevant searches and rank higher. Even moving up one rank can boost clicks by 30%+.
Product content excellence: 87% of consumers rate product content extremely or very important when deciding to buy. DSA automatically scans your product detail pages for completeness, consistency, and accuracy, alerting you to any issues that could turn off shoppers. With a single source of truth, keeping product content up to date across all channels becomes simple.
Competitive benchmarking: DSA sheds light on how your products stack up versus competitors in terms of search visibility, ratings and reviews, price, promotions, and more. By keeping a constant eye on the competition, you can quickly adjust tactics to win more sales. For example, if a rival lowers prices, you‘ll know right away and can decide to match.
Review monitoring and management: 91% of consumers trust online reviews as much as personal recommendations. DSA gathers all your products‘ ratings and reviews in one place so you can easily track average star ratings, identify common themes and complaints, and respond to negative feedback before it snowballs. You can also compare your review profile to competitors‘ to spot opportunities for improvement.
Sales and market share tracking: Estimating market share in ecommerce can be difficult since retailers rarely share sales data with brands. But DSA can provide a reliable proxy by measuring metrics like Share of Search (how often your products show up in search results vs. competitors) and Share of Reviews (what percent of reviews your products capture in a given category). Monitoring these can help you spot market share swings.
Inventory optimization: Few things frustrate online shoppers more than out-of-stocks. DSA tracks inventory levels across all your retailer partners, allowing you to forecast demand more accurately and proactively allocate supply. The result? Fewer lost sales and happier customers.
The unifying thread here is automation through data. By continuously collecting and crunching competitive metrics that would be impractical to track manually, DSA empowers brands to take timely actions to minimize threats and seize opportunities as they emerge. Over time, this drives significant gains in online sales and market share.
DSA in Action: A Conversation with e.fundamentals
To bring the power of DSA to life, we spoke with Greg Urquhart, CTO, and Jordan Dick, Director of Engineering at e.fundamentals, a leading ecommerce analytics platform serving large CPGs like Pepsico and Unilever.
Can you walk us through how your platform collects and uses public online data?
Jordan: Our web crawlers visit over 450 retailer websites daily, collecting data on more than 1.5 million products. We‘re essentially taking a daily snapshot of the entire digital shelf for our customers.
For each product, we‘re extracting data points like title, description, images, price, number of reviews, and search ranking for relevant keywords. We‘re also tracking competitor products in the same categories.
All of this data flows into our proprietary knowledge graph, where it‘s combined with customer data like sales and inventory levels. Our analytics engine then looks for anomalies and opportunities, which get surfaced to users in our web app.
The key is that this is all happening automatically and continuously. There‘s no way you could manually gather this breadth and depth of data across so many sites at this frequency. That‘s what makes the insights so timely and powerful.
What kind of results have your customers seen using DSA?
Greg: The impacts have been truly transformative for many of our CPG customers.
One large food brand was able to add an average of 20 long-tail search terms to more than 100 products. This massively boosted their search visibility and led to a 30% jump in add-to-cart conversions within 3 months.
Another CPG used our platform‘s content auditing tools to identify and fix thousands of product pages with missing or outdated information. Getting all their content buttoned up across channels led to a 25% uplift in conversion rate.
We also helped a snacks brand completely revamp their competitive positioning strategy based on insights from our market share and SOV dashboards. They were able to steal significant share from rivals and record 40% online sales growth.
When CPGs can continuously track and optimize their entire online presence, the revenue gains can be game-changing. It‘s not uncommon for our customers to see 30%, 50%, even 100%+ online sales growth.
What advice would you give brands considering DSA?
Jordan: First, don‘t underestimate the importance of data quality. Your DSA platform is only as good as the data fueling it. Make sure your vendor has rock-solid data collection tech and processes – it‘s the foundation everything else rests on.
Integration is also key. DSA becomes vastly more powerful when you can combine it with internal data to get a holistic view of your online business. Look for a platform that plays well with your existing tech stack.
Finally, change management is critical. Adopting DSA often means rethinking deeply ingrained assumptions, processes and org structures. Leadership needs to drive that cultural shift and empower teams to run with data-driven decisions. Start with a plan for injecting DSA insights into every part of your ecommerce operations.
The Future of Digital Shelf Analytics
As the battle for ecommerce market share intensifies, DSA adoption is poised to explode. Brands that aren‘t leveraging public data to automatically monitor, optimize and defend their online presence 24/7 will be flying blind versus competitors that are.
Going forward, we expect to see DSA become the nerve center for CPG ecommerce decision-making and operations. Advances in web data collection, machine learning, and real-time alerting will make insights ever more granular, predictive and actionable.
We‘ll also see DSA extend further along the supply chain, helping brands coordinate everything from demand forecasting to inventory planning to dynamic pricing and promotions. Integrations with emerging retail media networks will open up powerful new ways to shape the digital shelf.
Zooming out, large-scale collection and analysis of public online data will be a key enabler of the broader direct-to-consumer (DTC) shift in CPG. As more brands launch their own DTC sites and marketplaces alongside retail partnerships, DSA will be the glue that helps orchestrate a cohesive ecommerce strategy.
Ultimately, DSA isn‘t just about driving online sales today, but building the capabilities needed to thrive in an ecommerce-centric future. CPGs that embrace it now will have a major head start. If you haven‘t already, the time to start leveraging digital shelf analytics is now – your online growth depends on it.
