I went into the pit, chose "boss fight", fought 4 of the nihils, didn't die, didn't leave. Yet I didn't get the achievement..
You need to have all four of the aggro'd and attacking you before killing any of them, while replaying the quest fight (so not just in the post-quest room with nihil spawns).