Siman 41
Presented here for study, not as a practical ruling. The Shulchan Aruch rules by Sephardi custom, the Rema's glosses by Ashkenazi custom, and the Kitzur follows Ashkenazi custom. For a practical question, ask your rabbi.
א הַנּוֹשֵׂא מַשּׂאוֹי עַל רֹאשׁוֹ, חוֹלֵץ תְּפִלִּין שֶׁל רֹאשׁ עַד שֶׁיָּסִיר הַמַּשּׂאוֹי, וַאֲפִלּוּ מִטְפַּחַת אָסוּר לְהָנִיחַ עַל הָרֹאשׁ שֶׁיֵּשׁ בּוֹ תְּפִלִּין, אֲבָל דָּבָר שֶׁדַּרְכּוֹ לִתֵּן בְּרֹאשׁוֹ, כְּגוֹן כּוֹבַע אוֹ מִצְנֶפֶת מֻתָּר: הַגָּה: וַאֲפִלּוּ הָכֵי אִם הוּא מַשּׂאוֹי כָּבֵד ד' קַבִּין וְהַתְּפִלִּין נִדְחָקוֹת צָרִיךְ לַהֲסִירָן (בֵּית יוֹסֵף):
1 Laws regarding one who is carrying a load and how he should conduct himself with his תפילין. And there is 1 Se'if: One who is carrying a load on his head should remove his head תפילין until he removes the load. And even a cloth is forbidden to place over one's head when he has his תפילין on. But something which is usually placed on the head, like a hat or a turban is permitted. Rema: Nevertheless, if the load is [as] heavy [as] four kav's and the תפילין are pushed [because of it], you are required to remove them (Beit Yosef).