Informal caregiving is linked to psychological stress. However, recent studies have suggested a protective association between informal caregiving and mortality among caregivers. We sought to test the association between caregiving and survival in the Komo-Ise study, a prospective cohort of community-dwelling residents aged 44–77 years living in two areas in Gunma prefecture, Japan. Caregiving status was assessed in 2000, and 8084 individuals were followed for ten years. All-cause mortality was ascertained from official registers. Using multivariate Cox proportional hazards models, we found no statistically significant overall association between informal caregiving and all-cause mortality for either combined sexes, (HR 0.97, 95% CI 0.79, 1.19), men (HR 0.98, 95% CI 0.76, 1.27), or women (HR 0.95, 95% CI 0.68, 1.34). The propensity score matched model also showed no increased risk of all-cause mortality across all caregivers, male caregivers, and female caregivers. The association with all-cause mortality was not observed regardless of the presence of support for activities of daily living (ADLs)/instrumental activities of daily living (IADLs) or the relationships to the care recipients. In subgroup analyses, informal caregiving was not associated with increased risk of all-cause death across subgroups for combined sexes, men, or women, except for increased mortality among female caregivers in the lowest-income group (HR 1.75, 95% CI 1.03, 3.00). An increase in the risk of mortality was not observed among male caregivers. In conclusion, informal caregiving did not increase mortality as a whole, nor for most subgroups, while some sub-groups such as women in the lower socioeconomic status groups may be vulnerable to the adverse health effects of caregiving.