The family caregivers of people with mental illness may internalize the public stereotypes into the affiliate stigma (i.e., the self-stigma of family members). This study aimed to compare the affiliate stigma across schizophrenia, bipolar disorder, and major depressive disorder, and to investigate potential factors associated with affiliate stigma. Each caregiver of family members with schizophrenia (n = 215), bipolar disorder (n = 85), and major depressive disorder (n = 159) completed the Affiliate Stigma Scale, Rosenberg Self-Esteem Scale, Caregiver Burden Inventory, Taiwanese Depression Questionnaire, and Beck Anxiety Inventory. After controlling for potential confounders, the hierarchical regression models showed that caregivers of a family member with schizophrenia had a higher level of affiliate stigma than those of bipolar disorder (β = -0.109; p < 0.05) and major depressive disorder (β = -0.230; p < 0.001). Self-esteem, developmental burden, and emotional burden were significant factors for affiliate stigma. The affiliate stigma of caregivers is associated with their self-esteem, caregiver burden, and by the diagnosis.