Ah, my dear Mr. Rust...
You are a fascinating specimen. If you ever feel something prick the back of your neck - don't be alarmed, I'm just tagging you for research purposes.
In my still relatively short lifetime I have enjoyed watching men (as a group) slowly awaken to a very conscious awareness of themselves as objects of desire. This is an inevitable and equalizing consequence of women seeking to be evaluated by other criteria. If women can be judged by actions and wealth, men can and will be judged on a physical level. The Balance Factor.
Men are still only partially socialized to think of themselves in that way - to have to measure themselves against a physical ideal in order to compete for mates. And it often makes them very uneasy. They are not yet trained peacocks.
My little sister is in college. A year or so ago, she said rather proudly, "I
only date guys with six packs." 50 years ago few girls cared if a man had a six pack - he was judged by status and charisma and could trade on a good mug if he looked nice in a suit. 6 packs just used to mean beer.
So when you're "showing off" (and I do think from what I've gathered about you, you might be showing off a little bit - perfectly understandable if you're willing to cop to it), in a sense, you are rendering
your companions naked. In sweatshirts, you're all "equal"...sexually neutral. But you're setting yourself apart, drawing attention to yourself and making them feel exposed - open to a different kind of judgement. Sexual objectification for men. It's catching on artifically, like wildfire, thanks mainly to media saturation - but it's not quite all the way there yet.
A couple thousand years of prudish Christianity relegated the ideal of male beauty and virility that was cultivated in anitquity to dark, dusty corners of museums. Thank women, or thank Calvin Klein, but it's back with a vengeance.
However, men are still socialized to think that they
shouldn't be
able to appreciate another man. On a subconscious and often conscious level, admiring another man - even innocently - is often perceived as homosexual desire and should never even be admitted privately. So not only are you threatening on a competitive level, you may also be threatening the sexual identity of some of your companions by forcing their train of thought to shift to an uncomfortable track.
What Bill M said is also interesting - kind of a combination of the last two ideas. They can
say all of those pretty boys are homos, but they know you aren't so again their comfortable perceptions are challenged.
Interesting stuff, at least to me. If my train of thought is chunky, I offer the sad excuse that my keyboard is reacting slowly tonight, and editing is becoming bothersome.

So anyway, I see this as a culture clash of sorts. I see you, from the attitudes you're expressing, as part of a new culture.Part of this may be that new culture(or VERY old culture) of male physical objectification vs. the old culture of non-visual status cues. Women have been doing this for years, but men are new to the game. The good thing is that they still do tend to have a better sense of humor about it.
Just for the sake of playing Devil's advocate, I'll de-sexualize the whole issue and take another angle.
I live in LA. Everyone wears flip flops everywhere. It's not that I find their feet a distracting turn-on or an eyesore in particular. I just don't need or want to see their fucking feet while I'm standing in line at the grocery store, or eating dinner at a restaurant. I don't care if they're comfortable. I just want them to get dressed and be civilized! Put some shoes on, dammit! It's 45 degrees and you aren't at the beach! THere's a time and a place...
So if you're the guy who rips his shirt off when it's 75 degrees and partly cloudy, or to run to the mailbox -
They may just be thinking, "put a shirt on, you wanker!"
Sometimes, if there's one shirtless guy in a crowd of generally good looking guys - I just assume he's the one that needs the most attention.
There may be and probably are many dynamics at work.
I do find it all rather fascinating. I suddenly want to smoke a pipe and go grab a butterfly net.
Yes...fascinating.