A general server is set up for many uses probably using apache.
If a server is just set up for one particular type of website e.g. WordPress then much of the caching, compressing content etc. is handled by the server with litespeed or nginx.
This means less for you to worry about on the site, but stricter rules to follow some plugins won't be allowed, less load on the server so sites can display quicker. Server operator can monitor what is happening on the server more effectively and hopefully apply corrections before anybody's site speed is affected.
I run sites on several server set ups, and if the site is optimised, although there would still be gains on a litespeed server, they would be marginal for many sites.
If you have optimised your own site content, run a caching plugin, lazy load etc. and monitor your site's performance and you are happy with the speed of your site then do not change.