For the student assessments, the mean values of the specific study programmes are classified in relation to their relative position to the average value of all study programmes. A confidence interval vis-à-vis the respective mean value is a decisive factor in terms of the allocation to a particular ranking group. In addition to the number of assessments, this confidence interval takes into consideration the homogeneity of the assessments within a single department.
For the student assessments, a 95% confidence interval based on the assumption of normal distribution is used. If the confidence interval for a subject at an HEI is completely above or below the mean value for all study programmes (determined on a national basis) then the programme is allocated to the top or bottom group; otherwise the programme is placed in the middle group. The results therefore ensure that there are substantial and statistically significant differences in the mean values of programmes belonging to the top group and those belonging to the bottom group.
In contrast to rankings from other organizations based on quartiles (i.e. top 25%, top 50%, etc.) or that allocate HEIs to "Top 10" lists, the CHE UniversityRanking procedure does not define the number of HEIs that can be allocated to each group. The size of each group can vary based upon the extent of the internal variance between individual study programmes and the variance between analysed study programmes. As an example, if for a given subject, the differences between individual HEIs are minimal, and the responses from the departments are very heterogeneous, then most HEIs will be in the middle group with only a few HEIs found in the top or bottom groups.
It may occur that study programmes with the same or similar mean values but confidence intervals of different sizes are sometimes allocated to different ranking groups if these programmes are borderline cases in terms of the medium group and top/bottom groups. As a result, in rare cases, it may occur that a study programme with a better mean value is placed in the medium group whereas one characterised by a "worse" mean value is allocated to the top group.
Such a ranking procedure that deviates from a purely mean value-orientated group formation process is quite unusual but not at all implausible: the orientation towards confidence intervals shows consideration of the homogeneity of the assessments with respect to content; technically, on the other hand, it gives greater credibility to the "real" mean (statistical) value of the population. From this standpoint, it therefore makes sense that in those rare cases, a slightly worse but more homogeneous assessment from students leads to a better position than a slightly better but more heterogeneous assessment from the students. In the later case while the mean is higher, the confidence of that mean being a statistically accurate representation is lower. A demonstration of this situation can be seen in the figure below.
Figure: Ranking groups based on confidence intervals
The rank groups have the function of giving only a rough orientation. Within the top and bottom groups there may be significant differences between individual departments. Conversely, it is also possible that there is no significant difference in the mean value between some departments of the middle group and some of those in the top or bottom group. However, if the average assessment of the subject as a whole is taken for standard, the rank group allocation allows for a reliable identification of “good evaluation” and “bad evaluation”. It is part of the nature of groupings/group formations that in the case of very insignificant mean value differences, even very small differences may decide the placement into the top/middle/bottom groups.
When considering the position of the confidence intervals in the error bar diagram below, it is clear that for example HEI7 is on average evaluated better than the overall mean value. However, the length of the confidence interval does not allow the HEI to be placed in the top group. This may be the result of too large a dispersion/statistical spread/mean variation of the assessments or by a rather small number of cases. In contrast, for the same reasons, HEI32 is not allocated to the bottom group, whereas HEI28, which has a better mean value, is placed in the bottom group. In this example, the top group includes HEIs 1 to 6 and 8, while the bottom group includes HEIs 26, 28, 30, 31, 33 and 34.
Example: Error bar diagram for overall study situation
The pdf downloads for individual subjects contain an overview of the total mean value for ten selected indices from the student questionnaires well as the related error bar diagrams.