Training Data Platform Buying Guide

As a fairly new category of software, there is no defined process for purchasing a training data platform. This guide will help you navigate this process, whether that’s through a formal RFP process or through a more informal purchasing cycle. It will cover stakeholders, evaluation criteria, and purchasing processes.

Vision AI & Training Data

Vision AI is a discipline of Machine Learning focussing on unstructured data. It can easily be broken down into two parts:

Model selection and hyperparameter tuning
Training data

Previously, research has been focused on model selection and hyperparameter tuning, but increasingly Google, Tesla, Facebook, and other top AI companies have focused on realizing gains from training data.

Google estimates that 83% of models fail because of poor training data management, but there are also significant performance benefits from experimentation with training data, and great AI companies focus significant time on this experimentation.

What is a training data platform?

A training data platform forms part of the modern MLOps stack. It should enable the team not only to scale their training data but also to run experiments on that data to realize efficiency gains.

The process of annotating data is the core functionality of a training data platform, but good training data platforms allow for rigorous QA processes and provide a dataset management module that can allow users to realize, and explain, gains provided by better training data utilization.

A training data platform should not be expected to provide new raw data or full end-to-end production AI. It is part of a broader machine learning stack, including raw data capture, training environments, hyperparameter tuning modules, and production hardware. As such, good training data platforms should have a flexible, open API to allow for easy integration into broader stacks.

Preparing your organization for a training data platform evaluation

Stakeholders

A rigorous training data platform evaluation should consider the concerns of four key stakeholder groups

Your Annotation workforce
Your Annotator Management team (sometimes can also be part of point 3)
Your data science/computer vision engineering team
Executive stakeholders

Before beginning an evaluation of training data platform providers, consult each of these groups. Find out what they need from a tool, what they want from a tool, and what they can’t do with the current system (or anything in particular they dislike about this).

Typical priorities and roles in the evaluation process for these groups can be broken down in the table on the next page (which is by no means exhaustive).

Benchmarks

For any software purchasing decision, there will be qualitative factors (e.g. quality of support, UI), but anything that can be measured should be, both against the status quo and against other players in the evaluation

A good approach is to pick a limited number of projects to test and to evaluate those against key measurable criteria

Speed of annotation
Accuracy of annotation
Speed of administration
End-to-end project time

These projects should be varied across annotation and data types (i.e. if you’re doing semantic segmentation of MRIs, and classification of x-rays, test both projects on the platform). You should gather the existing benchmarks across these areas.

Table 1: The roles in a training data management platform evaluation

‍

	Typical Concerns	Metrics that matter to them	Role in evaluation
Annotators	Is the tool easy to learn? Is the tool easy to use? Does the tool allow me to automate routine tasks? Does the tool allow me to dispense with non-core tasks?	Ramp time for annotators Speed of annotation at Month 1, Month 3, and Month 12	Annotators should be allowed to try the software to evaluate the UI and provide feedback
Annotator Management	Can I easily add new annotators or manage my existing workforce? Does the tool have different permissions/access roles and easy authorization Can I easily report on annotator performance? How can I minimize annotator downtime? Can I easily assign work and roles to annotators?	Annotator efficiency (how much data can they process in a given hour) Annotation quality and accuracy Speed of administration (effectively, end-to-end, how long does it take to manage a new project)	The Annotator Manager should be a core part of the evaluation process They should be a key stakeholder in testing the software and provide reports on annotator performance They should provide efficiency benchmarks for comparison
Data Science / CV Team	Does the tool have a robust SDK/API for easy use? Does the platform minimize my involvement in the annotation process? Does the platform allow me to run experiments and explain the link between training data and model performance? Can the platform scale with our growing ambitions? Is there good support and a performant platform?	Speed of administration (effectively, end-to-end, how long does it take to manage a new project) Accuracy of annotation Performance of Models	The CV Team typically lead the technical evaluation process. This should be not only an evaluation of ease of use of the platform itself, but also understanding how it would fit into the broader stack. They should spend time evaluating the dataset management and administrative functions of a training data platform to ensure it will minimize their time in the weeds of these processes, but allow for better AI
Executive Stakeholders	Will this platform save me money? Will this platform increase the performance of our models? Will this platform free up more of my top engineers’ time? Is the platform and our data secure?	Return on Investment Model performance SLAs & Security	The Executive Stakeholders should review the business case of the product to understand it's ability to affect their north star metrics and KPIs. They should easily be able to estimate the ROI of the solution and understand how it might enable their team to hit their targets

Typical Concerns

Metrics that matter to them

Role in evaluation

Annotators

Is the tool easy to learn?

Is the tool easy to use?

Does the tool allow me to automate routine tasks?

Does the tool allow me to dispense with non-core tasks?

Ramp time for annotators

Speed of annotation at Month 1, Month 3, and Month 12

Annotators should be allowed to try the software to evaluate the UI and provide feedback

Annotator Management

Can I easily add new annotators or manage my existing workforce?

Does the tool have different permissions/access roles and easy authorization

Can I easily report on annotator performance?

How can I minimize annotator downtime?

Can I easily assign work and roles to annotators?

Annotator efficiency (how much data can they process in a given hour)

Annotation quality and accuracy

Speed of administration (effectively, end-to-end, how long does it take to manage a new project)

The Annotator Manager should be a core part of the evaluation process

They should be a key stakeholder in testing the software and provide reports on annotator performance

They should provide efficiency benchmarks for comparison

Data Science / CV Team

Does the tool have a robust SDK/API for easy use?

Does the platform minimize my involvement in the annotation process?

Does the platform allow me to run experiments and explain the link between training data and model performance?

Can the platform scale with our growing ambitions?

Is there good support and a performant platform?

Speed of administration (effectively, end-to-end, how long does it take to manage a new project)

Accuracy of annotation

Performance of Models

The CV Team typically lead the technical evaluation process.

This should be not only an evaluation of ease of use of the platform itself, but also understanding how it would fit into the broader stack.

They should spend time evaluating the dataset management and administrative functions of a training data platform to ensure it will minimize their time in the weeds of these processes, but allow for better AI

Executive Stakeholders

Will this platform save me money?

Will this platform increase the performance of our models?

Will this platform free up more of my top engineers’ time?

Is the platform and our data secure?

Return on Investment

Model performance

SLAs & Security

The Executive Stakeholders should review the business case of the product to understand it's ability to affect their north star metrics and KPIs.

They should easily be able to estimate the ROI of the solution and understand how it might enable their team to hit their targets

📥 Download: Training Data Management Platform Roles Overview

Table 2: Feature checklist and scoring

Every set of requirements is slightly different, but this should provide a good overall breakdown. A weighted priority score has been suggested, but can be adjusted. Some items, of course, will be breaking.

Feature	Weighting	Vendor 1	Vendor 2	Vendor 3
Data and Annotation Types
Privacy and Security
Annotator Speed and Efficiency
Quality Assurance
External Annotators
AI Models
Dataset Management
API & SDK
Support
Totals

📥 Download: Features Checklist

Download other charts

| 📥 Data Types and Annotation Types

| 📥 Privacy and Security

| 📥 Annotator Speed and Efficiency

| 📥 Quality Assurance

| 📥 External Annotators

| 📥 AI Models

| 📥 Dataset Management

| 📥 API & SDK

Conclusion

As discussed, every team has different requirements, and the tables above should be adjusted according to your own schema. Security should always be an essential requirement, but others can adjusted based on needs or complexity of needs. If you need any further assistance, please reach out to Matt Brown, matt@v7labs.com, who can help with any further clarifications.