Abstract: We present an algorithm of color image understanding which segments the image after analyzing sur-faces with color variations due to lighting condition and object colors. The work is based on dichromaticreflection model according to the strategy of hypothesis plus test, following the continuity of image and thefeature of color clusters, the algorithm completes the image segmentation and gives forth a physical des-cription of imaging process, including intrinsic images, segmented image, the color of light and objects. Re-flecting respectively the propertics of light condition and every objects, both matte image and highlight im-age compose the intrinsic images.