Well, it's just the way it was set up. On purpose. So the Senate is by state, period. That does make it less representative but that was the compromise (and it has some philosophical standing I think - our country is made up of states, it makes sense to do some national business and policy making as "states" with an equal number of people representing each state).
But as you say, the balance to that is the House - they represent the people in the states (in some sense). So the argument here is that for a very long time we expanded the number of reps based on the population, so each one represented a more or less stable number of people even as the total population of the country grew. But then we stopped doing that. As there isn't really a strong, clear reason to have stopped. You or I have much better access to a Rep if we're one of their 35,000 constituents than if we're one of their 750,000. We probably can't get back to 35k, but we can improve that ratio.
At some point I wonder about functionality, but my expectations for actual policy work out of the House is pretty low already.
I heard Levin discussing this on a podcast and he seems to think that's not a huge trade-off, because he doesn't expect much of that sort of thing from the House at all, so from a practical matter it's more about constituents feeling (and being) better represented than about getting them to pass better bills or anything like that (although there is a argument that you'd get more factionalism which might lead to more coalition building, which could lead to better law-making and functionality). I'm becoming a fan of his work lately (I think he was featured in W's
book last year). He has another practical election-related thing out recently, but it's paywalled at another "way to liberal to read op-eds from" site, so I won't bother linking (and due to the paywall I haven't read it myself, only exerts).