Emergency responders were called to the scene of a house fire in Madison early Wednesday afternoon.