Introduction: As the prognosis of early-stage breast cancer patients is excellent, prevention of radiation-induced toxicity has become crucial. Reduction of margins compensating for intrafraction motion reduces non-target dose. We assessed motion of the tumor bed throughout APBI treatment fractions and calculated CTV-PTV margins for breathing and drift. Methods: This prospective clinical trial included patients treated with APBI on a Cyberknife with fiducial tracking. Paired orthogonal kV images made throughout the entire fraction were used to extract the tumor bed position. The images used for breathing modelling were used to calculate breathing amplitudes. The margins needed to compensate for breathing and drift were calculated according to Engelsman and Van Herk respectively. Results: Twenty-two patients, 110 fractions and 5087 image pairs were analyzed. The margins needed for breathing were 0.3–0.6 mm. The margin for drift increased with time after the first imaging for positioning. For a total fraction duration up to 8 min, a margin of 1.0 mm is sufficient. For a fraction of 32 min, 2.5 mm is needed. Techniques that account for breathing motion can reduce the margin by 0.1 mm. There was a systematic trend in the drift in the caudal, medial and posterior direction. To compensate for this, 0.7 mm could be added to the margins. Conclusions: The margin needed to compensate for intrafraction motion increased with longer fraction duration due to drifting of the target. It doubled for a fraction of 24 min compared to 8 min. Breathing motion has a limited effect.