I like the idea of an unspoken romantic love between them. That from day one there was an attraction between them but there was an unspoken decision made to leave these feelings aside for the sake of their jobs. What I find more believable though and what I think Fellowes is probably hinting at is that they were simply colleagues. They got along very well and complimented each others work ethics. Then slowly, after all the years they worked together they started to care for each. Elsie in particular sees the world around them changing and I think realises better than Charles that they must look after each other because there will be no one else to do it. I think at this stage of their lives neither of them wants to be without the other. My favorite headcanon wants to believe this is because they are in love but my sensible head says it’s simply because they’ve grown used to each other.
I don’t think I explained that very well! :)