General framework for cross-validation of machine learning algorithms using SQL on distributed systems

摘要

A general framework for cross-validation of any supervised learning algorithm on a distributed database comprises a multi-layer software architecture that implements training, prediction and metric functions in a C++ layer and iterates processing of different subsets of a data set with a plurality of different models in a Python layer. The best model is determined to be the one with the smallest average prediction error across all database segments.