Caching is the act of storing information in hiding or for future use; in other words, a cached page or asset is a "snapshot" or "copy" of the page/asset that can be retrieved quicker than the live data. For websites built around a Content Management System, this helps lighten the load on your web server's resources.
Forms of Caching:
Below is a list of common types of caching that applies to websites (displayed in order of best to least performance boost).
Local browser cache (aka browser history)
Out-of-the-box, most web browsers store a cache/history of websites you visited. When you revisit a website you visit frequently, certain elements of that website and its webpages may be loaded from your cache such as the site's CSS (cascading stylesheets), background images, and some or all content. This type of caching minimizes the amount of requests made to the webserver and expedites the load time of the webpage.
[Web Server] In-memory cache
Some web-hosting providers such as Acquia and Pantheon as well as DNS traffic management service providers such as CloudFlare and Dyn provide the Varnish caching service; others allow you to custom install such services. These services utilize in-memory caching. In this form of caching, "snapshots" of a website resources and content are saved on the webserver itself. When this type of caching is enabled, you may find that content updates on the site are not reflected immediately to users who are not logged into the website, but pages tend to load faster than any that are setup to not have a cached version.
[Web Server] Database cache
Drupal has a built-in database cache that stores "snapshots" of frequently accessed resources like menu links, blocks and widgets, and static content. This cached data requires fewer database requests to load a page, but does not yield significant performance optimizations to a web page with several visitors at the same time.
A good example of where caching's effects are commonly observed: social media feeds. If you have a block showing throughout the website that features your recent Facebook activity, is it that critical for visitors to the website to see a live Facebook feed on your website? However, if you have a single page on the website dedicated to your social media feeds, it may make sense to not cache that one page.
As a rule of thumb, if your general cache settings are too frequent, then you're not really taking advantage of the caching features properly. You're better off adding some exclusion rules or cache regeneration rules for particular pages of content than having the majority of your content cached too frequently.