The chemical reactions and pathways involving phytosteroids, steroids of higher plants that differ from animal steroids in having substitutions at C24 and/or a double bond at C22. Phytosteroids are so named because they occur in higher plants; some, notably ergosterol, are also found in fungi.