Problem

Large datasets of unstructured data, when collected individually and uploaded to conventional cloud storage solutions, pose a time-consuming data organization process. Pair that with using different servers due to size limits, and data curation becomes a tedious effort. Working with owned data (such as images and videos of crops) allows computer vision models to be trained with highly relevant data, however, having a manual data pipeline can affect the cost and turnaround time to develop a viable product.

Solution

LayerNext’s MetaLake is an all-in-one repository for data management. Its advanced metadata capabilities allow data to be tagged as they are uploaded, whether remote or local, speeding up the data organization process and facilitating the ease of searching for specific data. The platform’s data visualization feature makes data curation and categorization effortless, even in large quantities. Automated workflows assign cleaned data to annotators, and ensure they are labeled and reviewed by internal specialists. These solutions allow the data engineering team to focus on monitoring annotated datasets, and achieving optimal model accuracy.

Result

By optimizing the data pipeline, utilizing automated workflows, and focusing on the quality and balance of data for model training, LayerNext enabled teams to work synchronously to deliver a viable product in less time and budget than expected, while speeding up the turnaround time to go to market.

An AgTech company explores the use of computer vision to improve yields and efficiency by integrating Computer Vision and AI in their mechanical harvesting equipment. The equipment would select and harvest mature crops autonomously, without causing unnecessary damage to plants or crops. To build out the AI, the company required software to manage a comprehensive data collection program, automated workflows to annotate large volumes of new data, and scalable model training for changing environments. Here’s how LayerNext met their needs:

Remote data curation and management

The project started off with gathering data. Technicians were out on farms using specialized equipment to take images and videos of crops and the environment around them. Heaps of large, unstructured datasets were uploaded to conventional cloud storage solutions on different servers, creating a time-consuming data organization process as the datasets grew. The AgTech company required a better solution and utilized LayerNext’s MetaLake.

Featuring advanced metadata capabilities and an all-in-one repository for uploading data, the technicians were able to tag datasets as they are uploaded. Automatic sorting and data visualization features sped up the data engineering team’s capability to curate, categorize, and search for specific data for model training. Built-in analytics on the LayerNext platform allowed the data engineering team to get an overview of the data collection process on demand.

Automated workflows for image annotation

As the success of an AI model is synonymous with the quality of the data and labeling efforts, the AgTech company was able to use LayerNext’s automated workflows to ensure that data is properly assigned, labeled, and reviewed by its internal specialists. Automating the project management aspect increased the productivity of the data engineering team, allowing them to focus on dataset maintenance.

Model training

For perennial crops, seasonal changes, weather circumstances, and farm locations are some of the factors that an AI model will have to account for. If the model were to take in all the data for training, it would be both expensive and time-consuming. Using LayerNext from the start of the data pipeline enabled the data engineering team to ensure that the dataset had good examples, and well-balanced composition for model training.

The data engineering team monitored annotated datasets and model accuracy using LayerNext’s Analytics. Using data and results from training and production environments to benchmark the performance of machine learning models, the team was able to view the high-level progress of the trained model and pinpoint specific issues that needed improvement.

LayerNext has the capacity to stand up to AI training workflows while providing precise insights into model training. By utilizing LayerNext, the tech company was able to save time, manpower, and related costs by taking the load off operational demands, such as remote data curation, data management, and workflow management. This strategy helped the company produce a viable product in less time than expected, speeding up the turnaround time to go to market.