I'm answering the more general question about EXIF data, rather than the specific question about the lens serial number.
It probably depends on how paranoid you are, and how much you have to hide (for example if you have a security clearance or are committing activities that might be illegal (or could become illegal in the future) or violate social norms, such as trespassing to take pictures inside an abandoned building). If you regularly post your activities on sites like Facebook, Twitter, or FourSquare, you probably don't need to be worrying about what the EXIF data in your photos might reveal. If you're more jealous of your privacy, you may care.
Time and date info in EXIF data can be an issue: someone mining your EXIF data could establish patterns (you go for a photo walk, away from your home, every Saturday afternoon; you regularly pass through an out-of-the-way place on your way to work/school/church; the length of your typical photo vacation could be calculated), or prove that you were at one location when you were supposed to be somewhere else (your boss or HR checking up on your activities when you call in sick; an insurance company questioning why you were on a moderately-strenuous trail when you're supposed to be resting after a workplace accident; a potential employer looking to see if you frequently take photos that would interfere with your working hours).
Someone could identify targets that carry lots of expensive photo equipment by looking at the range of focal lengths used or perhaps lens-identification in EXIF data, or by deeper analysis like, "this shutter speed at this time of day requires a tripod and filters, and since the wind was clearly gusting it must have been a high-quality tripod").
I'm sure that more sophisticated analysis (such as learning about your personality or habits based on commonly-used focal lengths, white balance, or other settings) could be done, too, but that's not something I understand well enough to be able to comment on.
If you regularly include EXIF data and then one time you don't include the EXIF data, that could indicate that you're trying to hide something.
If you're concerned enough about privacy or security that you want to strip EXIF data, you may also want to consider falsifying the EXIF data, which of course is a lot more work.