Whether you’re a retailer of a brick & mortar, an online, or a mixed store, for almost every retailer qualitative competitive data is like gravity. It’s the fundamental force for actionable analysis and KPIs outperforming.
Moreover, despite the growth of hyper-personalization trends which require even more qualitative data, a lot of retailers still use manual work or self-made web scrapers to collect and process competitive data. This approach leads them not only to increase in cost but to spend more time on validating and fixing data. As a result, such data can’t be used for decision making, regardless of whether the pricing is done manually by Category Managers or automatically with the help of an AI-based solution.
Alternatives of Competitive Data Providers
There are many data providers to choose from on the market. Some of them are simple data crawlers while others offer more intelligent data solutions with additional services and data quality warranties.
Just so that you are able to get a quick overview of the hardness of a choice, here’s a small list of providers you’ll probably come across on listings or in your Google queries results.
Read their official descriptions from their websites and try to get out of it whether you can rely on competitive data they serve or not:
Competera — The family of products for retailers to supercharge price formation with quality guaranteed data extraction and AI-driven price recommendations. Competera enables retailers to formulate optimal prices based on best-in-class competitive data and to outsource the whole host of retail sales results to algorithms processing any data affecting the margin points dynamic.
Profitero — E-commerce analytics, insights and strategic recommendations pinpoint how to improve daily performance across your sales channels. Eliminate manual collection of competitor price data with accurate daily competitor prices for branded and private-label products. Analyze your product assortment, prices, and promotions versus competitors, longer-term price gaps & trends.
Pricemanager — A revolutionary way to track your competitor’s pricing. PriceManager’s powerful, web-based reporting and analytics tool provides both online-only retailers and multi-channel merchants with meaningful, actionable pricing and competitive intelligence to help retailers with day-to-day price changes.
Metacommerce — An online platform for online stores to monitor prices, assortment and to analyze the information about competitors, allowing you to automate your pricing, product management, and marketing.
Proanalytics — We know everything about your competitors: price monitoring, competitor analysis. Rival price monitoring for the range of all your businesses and much more. All data is designed using your items.
Semantics3 — Product data solutions to power the next generation of e-commerce. Semantics3 operates the world’s largest e-commerce product database. We’re a trusted and reliable provider of ready-to-use structured e-commerce product pricing and metadata, with coverage on all of the top 800 internet retailers.
Revionics — We help retailers around the world gain that competitive edge by managing and optimizing their pricing, promotions, markdowns, and space. Created by retail experts, our cloud-based SaaS solutions deliver amazing year-over-year ROI with an attractive total cost of ownership using machine learning science that really works.
It’s hard to choose, right?
Down below you’ll find a detailed manual on how to intelligently select a competitive data provider and get information about any of the difficulties that you may encounter.
The Guide to Selecting a Data Provider
You can follow this guide when selecting a competitive data provider:
- Be sure that the product matches rate is compellingly accurate and manageable across broad sort of categories
- Define both the data scraping and delivery process benchmarks and gain commitments on their adhering from the provider
- Arrange a transparent monitoring environment for the matches and delivery metrics control
Although this may seem fairly easy at a first glance, it’s only the tip of the iceberg. Therefore, let’s get down to business and reveal the rest of it.
How to Ensure that Item Matches are Done Properly
A lot of data providers give no options for monitoring the quality of matches and to manage them.
At the beginning of the cooperation, it’s crucial for the retailer to set a pilot to test the data provider. This simple action allows you not only to check the quality of matches and how consistent the data delivery is but to see whether provider gives any tools to control it.
Otherwise, when monitoring a significant part of the range, even the slightest mistake in the matches will cause enormous losses.
In order to better describe what was just stated, let’s add up the number of pricing mistakes that a retailer can make with incorrect matches. Suppose the retailer sells 50K products and the accuracy of matches is 70%. As a result, about 15K products will be poorly priced.
| Accuracy of matches | Products affecting retailer’s margin |
|---|---|
| 70% | 15K |
| 80% | 10K |
| 90% | 5K |
| 95% | 2.5K |
That’s why it’s important to choose the right way to match products, or wisely choose a provider. In general, there are two different ways to match products: manual (visual) or automated matches. Both of them have their own advantages and disadvantages.
The main advantage of automated matches is their speed.
Machine learning algorithms are able to match 1K products every minute which exceeds the capabilities of the manual matching process. As a result, the speed of matching causes prices to decrease.
At the same time, automated machines come with every disadvantage associated with machine learning algorithms. Depending on both the data quality and the size of the data set; should the algorithm have been trained on a certain data set and then a data shift occurs (i.e., your rival alters the name of their products), the data shift point must be detected and the entire scope of products must be rematched with human involvement.
On the contrary from the automated ones, manual matches offer a fast launch, ease of scale, as well as capacity planning.
If created properly, they offer retailers much more accurate data but it’s more expensive at the same time. Additionally, in comparison to the automated capability mentioned early, manual matches are able to work with intricate industries where products contain lots of optional identifiers. Examples of this include perfumes, toys, apparel, etc.
The biggest downside to manual matches, aside from the price, is its speed. On average, the capacity of a matcher is about 600 matches each day. Also, the chance of a human-made error is always prevalent.
To get proper matches at the speed that the changes occur, a retailer must combine the best of both of those methods: establish the process of automated matching and then add onto it with the accuracy of manual matches.
For instance, a manual check can be used to qualify automated matching which enhances the matching algorithm as well as offers a higher matches quality.
With this kind of mixed approach, you’ll receive a high percentage of comparisons on many parameters as well as a small number of zero (not-found) prices with few errors within the data collected.
Which Data Crawling and Delivery Metrics Should Be Defined With a Provider as a Part of Service Level Agreement
For the majority of retailers, the quality is what’s complicated and non-transparent, but an urgent issue to estimate. It’s possible for data collection to be successful, however, the retailer will receive required data points that may not actually be relevant.
There are many cases when competitor sites guard themselves against automatic parsers (bots). In this scenario, the parser can receive distorted values such as prices without discounts, no-promo prices, etc.
If you want to steer clear of those kinds of cases, check up on whether your data provider needs to organize selective verification of the collected data, especially when you need to collect data from the websites that are always working on their security and constantly coming up with new ways to block crawlers.
The next indicator that impacts how effective pricing is: how fresh is the collected data.
Those industries that have both a high turnover, high customer elasticity and a low margin, such as electronics where rivals are changing their prices multiple times a day, it’s crucial to utilize the most recent market data. If not, you’ll fall out of the market in no time.
A Category Manager demands a clear understanding of when the data was actually collected to make the correct pricing decisions to utilize or get rid of certain pricing data.
Data Freshness, % = Data collected 2 hours before pricing / Scheduled data volume * 100%
Some data providers do not show that indicator, thus retailer gets data that was collected 10 minutes ago along with the data collected 48 hours prior repricing in the same array.
In order to let the manager make a smart competitive and efficient pricing decision, data must be as fresh as it can be and delivered within a small amount of time prior to when the category managers prepared to set the new prices.
| Data Freshness: | |||
|---|---|---|---|
| 48 + hours | 4-48 hours | 2-4 hours | 0-2 hours |
| Ancient data | Irrelevant data | Stale data | Fresh data |
The following indicator of data quality lies in the completeness which is the percentage of the data volume planned that actually ended up being collected and then delivered to the retailer.
Data Completeness = Collected volume / Scheduled volume * 100%
Every uncollected percent of data leads to thousands of products that are poorly priced.
If the volume of the collected data ends up being less than what it’s supposed to be, it’s crucial for the retailer to understand how important the missing data actually is:
- Is the data missing from either the market leaders or followers?
- The data from which product categories are missing?
- Is there any data about the KVI-products?
The data provider should give the retailer all of the instruments to control metrics mentioned above — matches quality, data freshness, and completeness — for every piece of delivered data.
How Do You Set Up Transparent Match Monitoring and Data Delivery Control System?
Information regarding whether or not the scheduled data collection has gone through successfully and how solid that data was needs be SLA-secured and shown to the retailer independently of the channel that they use (uploading .csv from the interface of the data vendor or reaching them via the API).
In this scenario, the Category Manager will always understand that, for instance, the TVs have been scanned successfully and there isn’t enough data about the small household appliances category.
Conclusion
Every single recommendation mentioned in this text should assist you in selecting a confident data provider. From now on, you can’t be confused by the buzz descriptions of numerous providers. Make a simple checkbox list and use it to ask the right questions while you’re searching for a potential partner.
You can obviously select whichever data collecting or pricing provider you like. You may even, as a beginner retail business, come up with a few reasons as to why you should choose the simplest one.
Regardless, we would recommend that you choose your data software wisely while keeping in mind a strategic picture of success:
- You need to choose a provider that delivers qualitative data from the beginning of cooperation: It’ll help you to grow quicker
- When you end up deciding to apply AI to your pricing processes, it’ll be much simpler to train ML models that have both clean and healthy historical data
To find out more about selecting a competitive data provider for price optimization, you may visit Competera on their website.
