Web Browsers have been built to render web pages with reference to W3C recommendations.
In the old days, web pages were typically built for Netscape Navigator and Internet Explorer.
But when web standards made at W3C then web pages of existing site start breaking because browsers could not start using that standard.
So browsers come up with 2 modes one for new standard-compliant sites and other for old legacy sites.
Now modern browsers come with 2 modes quirks mode and standard or strict mode.
Quirk mode is not any standard, when any web page does not have a Doctype then browser renders a web page in Quirk mode or quirk mode is turned on when there is no correct Doctype declaration in any web page and auto turned off after adding valid Doctype.
For internet explorer, if an HTML page contains valid <!Doctype> then IE uses one of the standard document modes and if there is not valid Doctype then it uses Quirk mode.
On internet explore, there is an option to change modes.
So, how browser know which mode to use – Browsers uses the DOCTYPE in a web page to determine whether to handle in quirk mode or standard mode. Always make sure that the web page should have valid Doctype to use standard mode.