Introduction: With the introduction of accelerated partial breast irradiation (APBI) and the trend of reducing the number of fractions, the geometric accuracy of treatment delivery becomes critical. APBI patient setup is often based on fiducials, as the seroma is frequently not visible on pretreatment imaging. We assessed the motion of fiducials relative to the tumor bed between planning CT and treatment, and calculated margins to compensate for this motion. Methods: A cohort of seventy patients treated with APBI on a Cyberknife was included. Planning and in-room pretreatment CT scans were registered on the tumor bed. Residual motion of the centers of mass of surgical clips and interstitial gold markers was calculated. We calculated the margins required per desired percentage of patients with 100% CTV coverage, and the systematic and random errors for fiducial motion. Results: For a single fraction treatment, a margin of 1.8 mm would ensure 100% CTV coverage in 90% of patients when using surgical clips for patient set-up. When using interstitial markers, the margin should be 2.2 mm. The systematic and random errors were 0.46 mm for surgical clip motion and 0.60 mm for interstitial marker motion. No clinical factors were found predictive for fiducial motion. Conclusions: Fiducial motion relative to the tumor bed between planning CT and APBI treatment is non-negligible and should be included in the PTV margin calculation to prevent geographical miss. Systematic and random errors of fiducial motion were combined with other geometric uncertainties to calculate comprehensive PTV margins for different treatment techniques.