Informal carers represent a substantial proportion of the population in many countries and health is an important factor in their capacity to continue care-giving. This study investigated the impact of care-giving on the mental and physical health of informal carers, taking account of contextual factors, including family and work. We examined health changes from before care-giving commenced to 2 and 4 years after, using longitudinal data from the Household Income and Labour Dynamics in Australia survey. The sample comprised 424 carers and 424 propensity score-matched non-carers. Health was self-assessed, measured with the SF-36 Health Survey Mental Health (MH) and Physical Functioning (PF) scales. Care-giving was classified as non-carer, low (<5 hours/week), moderate (5–19 hours/week) and high (20 or more hours/week). PF and MH change scores were regressed on baseline scores, care-giving, covariates (including work, family and socio-demographic characteristics) and interactions to identify impacts for subgroups. The physical and mental health impacts differed by gender, and care-giving hours and carer work hours were important contextual factors. Deterioration in both PF and MH was worse for females after 2 years and deterioration in MH was worse for males after 4 years. Among carers aged 40–64 years, there was a 17-point decline in PF (P = 0.009) and a 14-point decline in MH (P < 0.0001) after 2 years for female high caregivers working full-time and 9.3 point improvement (P = 0.02) for non-working male high caregivers. Change was not significant for non-carers. The study found that not all carers suffer adverse health impacts; however, the combination of high levels of care-giving with workforce participation can increase the risk of negative physical and mental health effects (particularly in female carers). Working carers providing high levels of care represent a vulnerable subgroup where supportive and preventive services might be focused.