Probably because they thought that it was unfair to allow men to vote for everyone in the population. For example, in this time period, a man would have to vote due to it being his civic duty, but the women, which theoretically would have made up half of America, would have no say.