Objectives: We aimed to describe the characteristics of caregivers with cancer compared to those without and analyze the association between having cancer and caregivers’ psychological distress in Japan. Methods: We used data from the Japanese Comprehensive Survey of Living Conditions in 2010, 2013, and 2016. The participants were 5258 family caregivers aged ≥40 years, caring for only one family member whose information in the dataset was available for all the covariates included in the model. The family caregivers’ psychological distress was defined by the Kessler Psychological Distress Scale (K6) score (K6 ≥ 5). We conducted a Poisson regression analysis to examine the association between having cancer and family caregivers’ distress. Results: The sample of family caregivers consisted of mostly females (69.3%) and people within the 40–64 years age group (51.8%). As a result, family caregivers with cancer increased across the survey periods; a higher number of participants were unemployed. When adjusted for covariates, including the presence of other diseases, having cancer was significantly associated with distress (risk ratio 1.33, 95% confidence interval 1.05–1.69) among family caregivers. Conclusions: Family caregivers with cancer are expected to increase in the future; it is important to provide them with more support in managing both their treatment and caregiving to cope with their distress.