SAFARI is a far-infrared grating spectrometer for the proposed Japanese SPace Infrared telescope for Cosmology and Astrophysics (SPICA). SAFARI will cover the wavelength range 34-230 um with a spectral resolution R~300 using four diffraction-grating modules populated with ultra-sensitive TES bolometers. These require a dark NEP < 2e-19 W/sqrt(Hz) to take advantage of SPICA's cold (<8 K) telescope. A Martin-Puplett interferometer can be inserted in the beam before the grating modules, providing a high-resolution mode with R varying from ~1500 at 230 um to ~11000 at 35 um. At SRON we have already fabricated TES bolometers that are twice as sensitive as required. Testing such sensitive detectors is challenging, particularly because a background power of only a few fW will saturate them. We have therefore built up a unique ultra-low background optical test facility based on a mechanically cooled dilution refrigerator, paying careful attention to magnetic shielding, EMI immunity, and light-tightness. We have verified that the optical efficiency (60%) of prototype detectors for SAFARI's short-wave band meets the requirement. We are now measuring the spectral response of prototype detectors using an external optical source and a Michelson interferometer. These measurements are important to ensure that we can measure the dispersed grating spectrum accurately. Furthermore, they allow us to investigate the effects of the intrinsic properties of the TES bolometers (e.g. load-dependent time constant and responsivity) on the extracted interferogram spectra. In this talk I will describe the SAFARI Detector Test Facility, highlighting how we optimized it for ultra-low background measurements, present the results of the optical testing, and discuss them in the context of the instrument performance