Making a Statistical Impact with Text Data

摘要

One of the great pleasures—and benefits— of belonging to the ASA is the opportunity to hear about problems on which we can make an impact. In this column and the next, I'll mention some of these, which will include progress on the 2019 initiatives (impact, disinformation, diversity), pointing us to new challenges to pursue. One big challenge to which statistics can contribute solutions concerns text data. Surveys and questionnaires prefer specific-choice questions (e.g., "On a scale of 1 to 10, rate ...") to avoid open-ended responses (e.g., "Other (specify) ...."). But in many areas, text responses may be preferable or even essential. For example, a physician may ask the patient to describe the level and location of discomfort or the times when pain is greatest. The data from such responses can be important for evaluating a treatment remedy under consideration (e.g., under what patient conditions is the treatment likely to be "successful," where "success" also must be quantified and may depend on the patient).