The Rema (OC 27:4) says that there is no problem of a Chatzitza between the Retzu’os of Tefilin and your skin only with the Batim of the Tefilin themselves. The Mogen Avrohom says there is a problem if there is an article of clothing separating them but not for a small thing. The Taz says that a chatzitza pasuls where the Retzuos need to be tied Min HaDin.
Rav Moshe Shternbuch (1:26) says that according to the Taz the only place a chatzitza is pasul is around the head and the first tying around the arm, but not all the wrapping around the forearm which is only a “Hiddur MiDivrei Kabala”. Rav Shternbuch says that nevertheless the Vilna Gaon, Pri Megadim and Kaf HaChaim all says that L’Chatchila one should make sure not to have any chatzitza anywhere under the Retzuos.
Rav Shternbuch conjectures that the reason is that if we allow a chatzitza under the retzuos of the forearm, people may think that there is no chatzitza by retzuos at all and only by the batim, which his wrong. Therefore we don’t have any chatzitza at all.
According to this, says Rav Shternbuch, even though Min HaDin wearing a watch is not a chatzitza, still we should take it off. However with regard to dirt on the arm, even Rav Shternbuch acknowledges that we need not remove it because of chatzitza. Revach

this is ONLY by Retzuos, But NOT with the batim. Look into the Mishna B. regarding how the achronim are bedieved mattir a dead lice under the bayis (as opposed to a live one or even a ‘gid’- some people use to hold the yud in place- which ARE a chtzita).
Yirei shamayim used to clean the upper arm even though they didnt notice anything prior to donning the tefilin shel yad…
Tshuvos Harashba , Teshuvas Harush and Pri Magaden. No reason at all to be machmur. Chatzitza by retzuos is only by the kesher…Ayin Shum. Do you tuck your retzuos shel rosh under you undershirt ?!?!? Again chatzitza only by the kesher.