This is one of those things we're told as we grow up. (I don't actually know if boys are told this. It seems like it might be told to girls only.) Don't wear white after Labor Day, which is the first Monday of September, in a way the official end of summer.
The whys of this are never explained to us either, but it's a fashion faux pas to wear white after Labor Day. This doesn't wear off until Easter.
It's been suggested that the reasons behind this have to do with New York fashion editors, who set the styles. In New York, where it gets cold in winter, white is going to blend in. This is also, I believe, the origin of the color 'winter white' which is off-white, so it gets around the rule.
A little digging on the Internet also got me an additional reason. Labor Day is like May Day, a day to celebrate universal workers' rights. It's a federal holiday in the US, meaning all government workers and schools have it off (a bank holiday, I think you call those here). Apparently this is the end of summer where only the rich can afford to go abroad and laze about in the tropics wearing white. So the workers kept to dark colors. This happened so long ago, though, that no one really knows about it. It's just passed down. No white after Labor Day. As a kid, I thought something terrible would happen if I wore white, and then as a teenager, I felt superior in knowing about this rule. My lowly classmates, in their ignorance, might wear white. I would sniff at them.