|
Late breaking results in intuitive experimental design
Jonathan D. Nelson
The talk will discuss new empirical and theoretical results on utility functions for assessing possible experiments' usefulness, in probabilistic information gathering tasks. Prominent functions in the literature include Bayesian diagnosticity, log diagnosticity, information gain (mutual information), Kullback-Liebler distance, probability gain (error minimization), and impact (absolute change). Key properties of these functions, and intuitive rationale to explain their behavior in several simulations, will be presented. The feature difference strategy, which has been observed in multiple experiments, is not merely heuristic (as has been claimed). Rather, I show that this strategy exactly implements impact. Previous empirical data do not establish which utility function best approximates human intuitions or behavior. However, new empirical data strongly contradict Bayesian diagnosticity and log diagnosticity. A theory of intuitive experimental design must address class-conditional feature dependencies in the context of sequential sampling, as well as differentiate between plausible utility functions for evidence acquisition. I'll suggest how behavioral and eye movement experiments can address these issues. |