Rain and snow seriously degrade outdoor video quality. In this work, a primary-secondary background model for removal of rain and snow is built. First, we analyze video noise and use a sliding window sequence principal component analysis de-nosing algorithm to reduce white noise in the video. Next, we apply the Gaussian mixture model (GMM) to model the video and segment all foreground objects primarily. After that, we calculate von Mises distribution of the velocity vectors and ratio of the overlapped region with referring to the result of the primary segmentation and extract the interesting object. Finally, rain and snow streaks are inpainted using the background to improve the quality of the video data. Experiments show that the proposed method can effectively suppress noise and extract interesting targets.