On the discussion of theoryhammer/playtest/non-random option, all of them are flawed in their own ways.
Theoryhammer only works with a lot of experience, a lot of math, and usually only works with a gods-eye-view. It doesn't take into account the press and flow of an actual game, where things (and players) function non-optimally.
Non-random is flawed in that it only works if the players don't take the lack of randomness into account. If they don't, then while it becomes a great intellectual exercise, all sense of risk goes out the window. Then there's also the issue of statistical determination. Let's assume you're playing with a 2+Init army. You activate 5 formations, knowing they can't fail. You know you're going to fail the 6th. So you pick one that is either currently useless, or one where it doesn't matter (A Shadowsword on an objective with LOS to it's target is going to be hit on a 2+ regardless of if it Sustains or not). This type of thing would have been most relevant in the first Rally phase. I failed to rally the almost pristine Crusaders on a 5+. Rerolled. Failed again. Rolled for my two remaining Armiger Sentinels on a 5+. Succeeded. That fits into the averages of statistics. If randomness weren't an issue, and I could pick the variance successes and failures, the game would have swung a lot differently.
Playtests, while they are technically the best option, are so fundamentally flawed that they often don't give meaningful advice. Firstly, the sample size is usually statistically insignificant. If 100-200 games were played, maybe things would even out. But those 100-200 games would have to be played by people that have both an equal skill level, or knowing whether the win was due to the balance of the list, or the skill of the players. Also, consistency is needed. Some players adapt quickly, and so early/late wins may be skewed by that. Others may have a particular tactical style (assault oriented, not enjoying/capable at primarily shooting forces). Then there's the random factor, which can completely skew it, as well as psychological impacts. There's so much variance, that playtesting is next to useless as a refined design tool, but they do make for good reading.
For the most part, design seems to be a combination of small amounts of Theoryhammer and Playtesting, with a large pinch of "F'it, it's close enough.". Because in a lot of circumstances, there's still wild gaps in opinion on established lists (Predators/Land Raiders as Transports/Shadowswords vs Baneblades/blah blahblah blah).
I'll continue to playtest, and I do want other people to playtest, but I'm not going to have playtesting impractically skew the design process. Or else Knights would be about 30% better/cheaper than they are now, and I just don't think that's justified. They wouldn't have to make Dangerous Terrain tests, for one (3 more Knights joined the ranks of the sleepy in that game. My opponent's DN's failed 0, despite making about twice as many). I mean, that's proof Walker on Knights isn't effective enough, yes?
Morgan Vening
- Knightworld Sub-Champion
