Mufti Ebrahim Desai gives an explanation of the phrase in the Quran dealing with female modesty, stressing that hijab is mandatory in front of male slaves but not female slaves:

Fatwa, posted 4.22.2010, from South Africa, in:
Religious Authority: 
Mufti Siraj Desai
Fatwa Question or Essay Title: 
Mufti Ebrahim Desai gives an explanation of the phrase in the Quran dealing with female modesty, stressing that hijab is mandatory in front of male slaves but not female slaves:

Question: Why is it allowed for Muslim females not to observe hijab in front of slaves (or the slaves whom their right hands possess according to the Quran) [particularly as it may lead to fitna if the slaves are young male/men?



Answer: According to the explanation of verse 31 in Surah Nur, great scholars including those from the companions and the followers have indicated that the part of the verse, "or to those owned by their right hands," applies only to female slaves, which means that hijab will be necessary in front of male slaves as it is in front of all other non-mehrams. Sa'id ibn al-Musayyib has advised not to be under the misconception that these particular wordings in the verse are general for both male and female slaves. It is only female slaves that are included in this verse. (Ma'ariful Quran, 6/415) Abdullah ibn Mas'ud (RadyAllahu Anhu), Hasan al-Basri (Rahmatullah Alayhi), and Ibn Sirin (Rahmatullah Alayhi) have stated that it is not permissible for a male slave to see his female owner's hair. (Ruh al-Ma'ani) Ibn Jarir (Rahmatullah Alayhi) also stated that this verse is referring to non-Muslim women from amongst the idol worshipers, not men. It will be permissible for a Muslim female not to observe full hijab in front of her since she is her slave-girl. (Tafsir ibn Kathir)