I believe you need to pick up more than five skulls.