The evaluation of a speech application consists either in a series of controlled tests of the application performed by a selected group of test users or the test of the application in an artificial test environment ('test bed') or both. Usually all tests are monitored (recorded) and later analysed with regard to pre-defined test criteria. Test results are interpreted and summarized in an evaluation report that might be the basis of strategic decisions of the customer (e.g. to use speech recognition in a product or not). Therefore an objective and independent system evaluation is of paramount importance.

Since the field of speech driven applications is rather young and such applications tend to be very different in techniques used and procedures applied, there exist no recognized evaluation standards for speech applications. Each case must be analysed very carefully to determine those test criteria that really enable the customer to come to a proper judgement of the new technology.
Walker et al (1997) have proposed the PARADISE framework to evaluate spoken dialogue systems. But unfortunately we found in several evaluations that PARADISE is only applicable for a very simple and spezialized form of dialogue system.
For the case of multi-modal speech applications the BAS has developped another evaluation scheme called PROMISE (Beringer et al. 2002) which was successfully applied in the evaluation of the SMARTKOM systems.