Family caregivers of people with dementia (PWD) have a heavy care burden. Affiliate stigma is the stigma internalized by individuals associated with PWD. Limited research has addressed the affiliate stigma among caregivers of PWD and its influence on caregiver burden. Thus, our study investigated the burden of caregivers of PWD and its relationship with affiliate stigma. In addition, we examined the factors related to affiliate stigma. This cross-sectional study was conducted in a general hospital in Taiwan. We recruited 270 PWD and their family caregivers from the outpatient department. Relevant demographic and clinical assessment data of the patients and caregivers were evaluated. Regression analysis was performed to examine the factors associated with affiliate stigma. In total, 23.7% of the family caregivers had depression and 37.4% had anxiety. Male caregivers had higher levels of anxiety and heavier care burdens related to affiliate stigma compared with female caregivers. Moreover, characteristics such as younger age and low levels of dependence in daily activities among PWD were associated with increased affiliate stigma. A higher family caregiver burden was related to more severe affiliate stigma. Interventions for decreasing the family caregiver burden might reduce the effect of affiliate stigma.