It seems to me that you should be allowed to remove you excess body hair with a scissor or electric shaver due to a combination of factors in Halacha, but not with a razor.
Shulchan Aruch YD 182 writes that a man may not remove the hair from his armpits and pubic area even with a scissors, but from other parts of the body it is only forbidden with a razor. However, the Tzemach Tzedek extends the prohibition of cutting with a scissors even to the face, as you mentioned. Additionally, Shulchan Aruch writes that if it becomes common for men to also remove their body hair it is no longer prohibited because of lo silbash, and the Prisha extends this to a situation where goyim remove the hair and Jews do not while other seforim disagree. This would seem to be the circumstance here, because it is common for many goyim to shave their chests. Furthermore, Shulchan Aruch stipulates that issur is only when shaving for beauty and not for health issues. While Shalom Bayis and "feeling dirty" may not fit into a strict definition of "health issues", there is a fascinating teshuva in Shoel u'Meishiv on the topic. There was a young man whose hair suddenly tuned white on one side of his head and face and he suffered extreme anguish and embarrassment from the condition. While other Poskim disagreed, the Shoel u'Meishiv permitted him to dye the white hairs black.
So it seems to me that while none of these considerations by themselves are conclusive, the combination should permit you situation as long as you don't use a razor which creates the potential of an issur d'oraisa.