Abstract: To effectively assess the property and efficacy of the numerous content-aware image resizing methods, a new subjective quality assessment method is presented in this paper.The method involves three part works.The first, the user test experiment scheme of combining the incomplete paired comparison and a randomized block design is proposed.Because the incomplete paired comparison design does not ask a participant to compare all pairs based on the experiment image set, this scheme can reduce the impact due to the test subject attention instability.Moreover, this scheme tries to minimize the discrepancy of the pairs compared by the different test subjects by the randomized block design.The second, the circle errors in the experiment results can be checked in real-time.Once a circle error is made, the test subject is asked to confirm his previous selection results to reduce the error results due to the thoughtless.The last, many non-parametric statistic methods are used when experiment data is analyzed.Specially, check the experiment data and see if the results selected by multiple test subjects on the same comparison pairs have same agreement preference.Once the agreement preference is salience, the effect of the different content-aware image resizing methods is sorted and the methods are grouped by combining the Friedman test, the Page test and the HollanderWolfe comparison method.Based on the above experiment scheme and data analysis method, the large scale user study is conducted to assess a number of the representative content-aware image resizing methods.The superior coefficient of agreement from the experiment data shows that our experiment results are supported by the major test subjects.