Learning page-independent heuristics for extracting data from Web pages

摘要

One bottlenck in implementing a system that intelligently queries the Web is developing `wrappers' --programs that extract data from Web pages. Here we describe a method for learning general, page-independent heuristics for extracting data from HTML documents. The input to our learning system is a set of working wrapper programs, paired with HTML pages they correctly wrap. The output is a general procedure for extracting data that works for may formats and many pages. In experiments with a collection of 84 constrained but realistic extraction problems, we demonstrate that 30/100 of the problems can be handled perfectly by learned extraction heuristics, and around 50/100 can be handled acceptably. We also demonstrate that learned page-independent extraction heuristics can substantially improve the performance of methods for learning page-specific wrappers.