Dust and Scratches Filter
For dirt and smudges, if your background is untextured you can use Filter > Noise > Dust and Scratches. This will remove any defects, of any size you choose, in one go. I would make a rough selection of the background, put into a new layer, run the dust and scratches filter. Then you have the ability to use a mask if you want to be more selective about where you apply, and don't apply, the effect. If you make the layer a smart object you can revise the filter settings if you have over- or under-done the filtering.
Healing and Patching
For larger patches, use the healing tools (particularly the patch tool). Select an area with a defect, then drag that to a good area. Photoshop will fill the defective area with something that has nice texture and color.
An alternative is to select an area that is good (no defects) and duplicate that into a new layer. Move that over the area that has problems. Cover the new layer with a black mask, then paint in with white, at low opacity, where you want to cover up the defects. This means you end up with all texture and light variations, rather than painting a solid color that will look artificial. You'll need to use an area that closely matches the color and brightness (or apply a levels/curves adjustment to make it match)
Content-Aware
Another simple method is to select an area, then Edit > Fill and select Content-Aware. This will remove the defect and attempt to match color, brightness and texture from surrounding areas. This works well in the interior, but may have problems near edges.
If you have wrinkles or folds that are not close to the subject, you can use content-aware scaling to stretch the background (leaving the subject unaltered) so that the areas containing the defects are dragged out of frame.