In the circuit above, the two capacitors highlighted with the red background perform decoupling and noise-reduction functions.
In DC electronics, capacitors can be used as a source of energy, and as a filter.
In the first instance, a capacitor can be used as a reservoir of energy that can substitute a battery of a power supply in situations where the circuit needs more current than what the main sources can provide. There is a particular class of capacitors, “super-capacitors” that are used in the place of batteries for devices that work in environments where a battery may not do too well (like in very hot places).
In many DC applications, you can add a capacitor across the Vcc the Vcc and GND lines to smooth out noise from the power supply. This is not so important if your circuit is powered by a battery (unless your circuit contains fast-switching components, like a microcontroller), but it is important if your circuit is powered from an AC/DC power supply. This kind of capacitor is called “decoupling”. You typically find very small-capacity capacitors here, around 0.1uF. They are not meant to store energy, but to filter out this external power noise.
But noise can also come from within your circuit. When your circuit contains fast-switching components, like an Atmega328p, these components can draw current from the power supply that changes rapidly. These tiny changes in the current draw can cause tiny fluctuations in the voltage supplied. A capacitor connected to the microcontroller’s GND and Vcc pins, as close as possible to those pins, will help smooth out those fluctuations.
So, in practical terms, when would you use a bypass/decoupling capacitor?
If your circuit contains a microcontroller or something similarly fast switching, then always include a small ceramic decoupling capacitor connected very close to that fast component’s Vcc and GND pins. This will take care of the noise.
If your circuit contains a component that can, occasionally, draw a lot of current like a motor or a transmitter, then also add a larger bypass capacitor.
This Atmel guide recommends that you include decoupling capacitors to all Vcc-GND pairs and also provides the appropriate values and locations (on the PCB) for these capacitors. The schematic above is from page 6.