In “The House We Never Built,” Gabrielle Aplin is expressing her feelings for her old lover in a metaphor. “The House” Gabrielle refers to throughout the song symbolizes her relationships with her ex-lover.

After both she and her lover abandoned the house, she is left to wonder what would have happened if they had both stayed. What could they have done together? It isn’t what’s left behind that hurts her, its what they could have built if they had stayed. What would have been of the house they never built?
As she looks throughout the house and sees all the unfinished projects, she feels the need to burn whatever is left of the house to the ground.

Nobody is going to go back to admit that is over. The bricks have gone tired of waiting and the windows have shattered and nobody is no longer welcomed. In order to move on, she must burn down the house they never built.