Recently, I was working on a feasibility study of measuring/detecting IM3s of an ADC output by capturing only 512 samples, where the sampling rate is about 300MS/sec. During this research, I came across "
On the Use of Windows for Harmonic Analysis with the Discrete Fourier Transform", by F. Harris (1978). This paper is a must-read for signal processing folks.
The concepts such as spectral resolution, the window processing loss (gain), 6-dB BW of the window, etc. need to be considered in any DFT design/analysis. As the figure below, with 512 samples, the signal intermods are not detectable if we do not use a proper window.
