Learning algorithm evaluation is usually focused on classification performance. However, the characteristics and requirements of real-world applications vary greatly. Thus, for a particular application, some evaluation criteria are more important than others. In fact, multiple criteria need to be considered to capture application-specific trade-offs. Many multi-criteria methods can be used for the actual evaluation but the problems of selecting appropriate criteria and metrics as well as capturing the trade-offs still persist. This paper presents a framework for application-oriented validation and evaluation (APPrOVE). The framework includes four sequential steps that together address the aforementioned problems and its use in practice is demonstrated through a case study.