The benefits of neural approaches are undisputed in many application areas. However, today's research practice in applied machine learning—and in particular in recommender systems research—can make it difficult to understand what represents the state-of-the-art in a field and how much progress is achieved through novel technical approaches. The underlying reasons are that researchers often use a variety of baselines, datasets, and evaluation procedures to demonstrate progress beyond the state-of-the-art. With this work, we aim to contribute to a better understanding of what represents the state-of-the-art in the fast-developing area of session-based recommendation and to what extent neural approaches help to achieve progress in this field.

To that purpose, we have conducted an extensive set of experiments, using a variety of datasets, in which we benchmarked four neural approaches that were published in the last three years against each other and against a set of simpler baseline techniques, e.g., based on nearest neighbors. The evaluation of the algorithms under the exact same conditions revealed that the benefits of applying today's neural approaches to session-based recommendations are still limited. In the majority of the cases, and in particular when precision and recall are used, it turned out that simple techniques in most cases outperform recent neural approaches. Our findings therefore point to certain major limitations of today's research practice. By sharing our evaluation framework publicly, we hope that some of these limitations can be overcome in the future.