While some of it can be cultural. I suspect most of it is from personal experience if your favourite childhood toy was blue or your bedroom was green thats going to be a significant influencer on emotional associations you have with certain colours.
I don't think any serious thought can be attached to notions that people respond unvirserily to certain colours a certain way.
Isabelline has an interesting story. The name of Isabelline, this grayish-yellow colour comes from the colour of the underwear of Isabel Clara Eugenia, daughter of King Philip II of Spain who at the siege of Ostend vowed not to change her underwear until the city was captured. The siege lasted for three years so the colour of her underwear must have been truly grayish-yellow (isabelline) when the city was finally taken.
Capt'n Midnight wrote:
In Irish the word for pink is bándearg "WhiteRed"