Procedural level generation has long since been prevalent in video games. A desire to make the methods used more generalised has recently sparked an interest in adapting machine learning for this purpose. However, this field is still relatively nascent and has several open questions. As such, this study investigated several metrics for the evaluation of machine learning assisted level generators for Super Mario Bros. This was done by using a generative adversarial network (GAN) together with evolutionary programming to generate levels that maximize the aforementioned metrics individually. Then, in order to establish correlative relationships, a user study was conducted. In this user study, participants were asked to play through the generated levels and rate them according to enjoyment, aesthetics, and difficulty. We show significant correlations between several metrics and the three dimensions of quality; some such correlations are also, seemingly, independent of prior gaming experience. We contribute to the field of machine learning assisted level generation by 1) reinforcing certain metrics’ validity for use in the evaluation of level generators, and 2) by demonstrating that this evolutionary approach can be used to control difficulty effectively.