Widget detection on screenshots using computer vision and machine learning algorithms

摘要

In this paper we consider the problem of detecting and recognizing widgets in screenshots of computer programs’graphical user interface (GUI). This problem is fundamental in business process automation. The solution wepropose here is based on detecting GUI elements with Canny edge operator, and recognizing already detectedGUI elements with classifiers: neural networks, random forests, XGBoost, and others.