How do you stay calm when a fight breaks out?

First off you should assess how and why you keep pissing people off that "it happens all the time".
Beyond that you seem to be saying that you become paralysed in the face of violence. This is pretty normal if you've had a secure upbringing. The way to get over it is to immerse yourself in full contact training sanshou, Muay Thai, MMA whatever.
A lot of people who walk through my door are the same at first, they flinch at strikes close their eyes etc. Often if they've come from previous light contact or make believe arts they aim to miss fall short etc which is useless to them and their training partners. At first I sometimes see them look a bit horrified at the training. After a few belts they usually out of revenge strike back this is good it allows them to get over their virgin attitude to violence. After about on average 3 months of classes with drills and sparring they have totally transformed their attitude to violence it's no longer shameful or to be feared it's an art. Draws and faints, timing, structure and angle, integrating punching, kicking and throws. Developing a fighters hard and resourceful intent, the worry is not being hurt it's being caught with a lucky shot.
At that stage you'll find too that you're not interested engaging in conflict with washed up skill less chumps from "the street", you're more into focusing on taking apart an international champion, pretty life changing overall!

Watch out for what I’d call “ape-sh1t” kung fu, there are those who’ll advise to use your fear and anger and simply lash out. If you do this abandon any hope of fine work neatly done, you’ll have no guard or awareness of holes in your defense or see any targets, no environmental awareness (not know where you are, or walls, or footpaths etc....i.e. tunnel vision) and you’ll look like a tard.
And remember shouting only scares those who’ve had a comfortable life, i.e. those less likely of being a threat, like wise flailing around. When it counts when you’re not facing handbags or bean curd boxing, when it counts, when in the ring, or if unfortunate enough “on the street” against someone plenty accustomed to violence and intent on dishing it out, then going ape-sh1t will leave you knocked the f out or worse, as even a child could drive a truck through your arms.
It happened to me twice only competitively, (dumbness gets weeded out pretty quickly on the international sanshou scene) where the opponent gave up and went all out ape-sh1t, both times I knocked the opponent out, once so badly as he was rushing and flailing at me, I simply stepped back from side to side, added a weave, (we call it seven star step) and caught him with a left uppercut which left him gibbering in a seated position on the Lei Tai. He was still gibbering for 10 minutes after being taken off on a stretcher. He recovered in hospital, but I got to say I was worried. Looking back I still see what he did as one of the dumbest moves of all time. The only better example was witnessing a friend of mine Nick Evagorou of Nam Yang dislocate the knee joint of another would-be sanshou fighter, I don’t remember the guys name, he was new and we never saw him again. Basically he too went ape-sh1t with frustration, and tried some Jet Li move with three kicks, Nick caught his leg, twisted and threw, but the dumbass had already begun his next kick, being on ape-sh1t auto pilot he couldn’t help but therefore add to the twist, it was before the days of YouTube, but I can still hear the pop.