The application of mineral and organic phosphorus fertilizers to arable land has greatly increased crop yield to meet the world food demand. On the other hand, impurities in these fertilizers, such as heavy metals, are being added to agricultural soils, resulting both from the raw materials themselves and the processes used to obtain the final product. Cadmium, a non-essential and toxic heavy metal, has been found in relatively high amounts in common P fertilizers obtained from sediments. This metal poses a high risk for soil fertility, crop cultivation, and plants in general. Furthermore, human health might be compromised by the cadmium concentrations in agricultural and livestock products, due to the bioaccumulation effect in the food web. The accumulation in the different matrixes is the result of the high mobility and flexible availability of this harmful metal. This review summarizes risks to human health, the factors influencing cadmium movement in soils and crop uptake, as well as common plant responses to its toxicity. In addition, it summarizes cadmium balances in soils, trends, long-term experiments, and further studies. Cadmium inputs and outputs in arable soil, together with their calculated concentrations, are compared between two different regions: the European countries (in particular Germany) and China. The comparison appears useful because of the different proportions in the inputs and outputs of cadmium, and the diverse geographical, environmental and social factors. Moreover, these variables and their influences on cadmium contamination improve the understanding of the pollution from phosphate fertilizers and will help to establish future mitigation policies.