Assuming you're using Windows, part of the problem is that there is no "Date Taken" tag. Windows fills this property with different metadata, depending upon what it can find. The mostly likely tag (and the one Windows gives highest priority to) is the EXIF:DateTimeOriginal tag. So the best command for you to try would be:
ExifTool "-FileModifyDate<EXIF:DateTimeOriginal" DIR
Edit: I made a more complete command. This will set the FileModifyDate by trying all the various metadata Windows uses for the "Date Taken" property, in order of priority. It uses the fact that when ExifTool has two assignments that affect the same tag, the latter takes precedence. From what I've discovered, the XMP:DateTimeOriginal has the least priority, so it's used first, while the EXIF:DateTimeOriginal has the highest priority, so it's used last.
ExifTool "-FileModifyDate<XMP:DateTimeOriginal" "-FileModifyDate<EXIF:CreateDate" "-FileModifyDate<XMP:CreateDate" "-FileModifyDate<$IPTC:DateCreated $IPTC:TimeCreated" "-FileModifyDate<EXIF:DateTimeOriginal" DIR
exiftool -time:all -a -G0:1 -s yourfile.jpg
and give the exact "date modified" tag (and group) you want to change? \$\endgroup\$