Metal nanomaterials have been extensively investigated owing to their unique properties in contrast to bulk counterparts. Gold nanoparticles (e.g., 3-100 nm) show quasi-continuous energy bands, while gold nanoclusters (<3 nm) and complexes exhibit discrete energy levels and display entirely different photophysical properties than regular nanoparticles. This review summarizes the electronic dynamics of these three types of gold materials studied by ultrafast spectroscopy. Briefly, for gold nanoparticles, their electronic relaxation is dominated by heat dissipation between the electrons and the lattice. In contrast, gold nanoclusters exhibit single-electron transitions and relatively long excited-state lifetimes being analogous to molecules. In gold complexes, the excited-state dynamics is dominated by intersystem crossing and phosphorescence. A detailed understanding of the photophysical properties of gold nanocluster materials is still missing and thus calls for future efforts. The fundamental insights into the discrete electronic structure and the size-induced evolution in quantum-sized nanoclusters will promote the exploration of their applications in various fields.