Living alongside and working with people with dementia who yell or strike out can be very demanding and sometimes harmful. It is generally understood that such actions may be a response to the social and physical environment, yet very little attention has been paid to understanding what role the environment plays in eliciting and responding to these actions across different settings. Drawing on 27 semi-structured interviews with formal and informal carers, this article examines how carers understand the actions of people with dementia in relation to their environment. We identify individual, interpersonal, local environmental, macro-scale and temporal dynamics that influence the actions of people with dementia and carers' capacities to work with them. Drawing on relational thinking and the concept of structural violence, we argue that many of the problems and solutions associated with aggressive actions are located outside the person with dementia in broader relations and systems of care. We conclude with a discussion of how these understandings can inform developments in the systems of home and community care.