I think you hit on something there. While there are other, well documented, issues at play here that leads to the perception of an TCU attendance problem (smallish liberal arts college, predominantly female student body from outside the football hotbed of Texas, smarter-than-the-average-bear student body that isn't particularly into sports, the heat, etc), the attractiveness of the opponent is a major player in all this and attractiveness can't just be measured by poll rankings.
What if we were playing San Diego State yesterday instead of West Virginia? San Diego State is ranked higher than West Virginia but does anyone believe for a second that more fans would have come to see them? The answer is no because there really isn't that much interest for a TCU fan to see either West Virginia or San Diego State! When the opponent is someone that TCU fans "care about", for whatever reason, the crowds do show up (see the old TCU/Utah and TCU/ BYU MWC matchups).