AJediKnight
Hey there!
I am a little curious about the decision to use Max Diffs instead of a rank ordering question when you only have 4 items per set. Rank ordering is normally fine until you have seven or more items, and would offer more detail. Was it just a matter of your initial question being more focused on extreme preferences? Maybe you already had a vague estimation of preferences (which seems likely), and designed the survey moreso for clarification?
Hey there!
Asking players to rank-order a set of 4 questions 12 times (or even like 6) is a HUGE cognitive ask! It probably would take them half an hour to have filled out the survey. We try to keep our surveys shorter, when possible, to respect player time.
Also, Max Diffs work either by selecting the "most" option, or the "most" and "least" option. Sawtooth software doesn't currently offer a "rank the full set" option, although that would theoretically reduce the amount of screens we have to show. We use the "most" and "least" because it gives us the most data possible.
As for the vague estimation of preferences, we know that we (as Rioters) have a super different experience of the game than players. This is especially true of PIE, the team who works on things like Skins, Loot, Event Passes, and other in-game products. We want to make sure we don't bias the results by assuming our understanding is representative of player experience.
Thanks for the question :)
Read more