It sounds like you've built up resentment already so it won't end good.
Just alter your plans for a while and he'll stop expecting.
Leave earlier or later a few times. Go shopping after work etc.
Anyhow with CoVid now you have the ultimate excuse to get him off your back. Tell him you don't feel comfortable not abiding by the 2m rule.
These arrangements get messy, not just the financial aspect of it.
There's the insurance issue for starters.
There's also the feeling of being tied to his routine and not being able to do your own thing of you want to make plans.
There are however people out there who will never get you even a bottle of wine once a year after you give them a lift for a year.
One of those I guess since the op brought the issue up.
COVID 19 is your friend this year. Perfect excuse
It's nice to be nice. Some will take advantage for years if they can, though.