It is primarily the difference in the varying subject distances and magnification ratios and how they interact with the resolution limits of the lens.
The effects of diffraction and sensor blooming, though measurable under laboratory conditions, are more subtle. If an image is taken below the diffraction limited aperture for a particular sensor and there are no fully saturated pixels then those effects will be demonstrated much less in the resulting image than if it is taken at an aperture above the DLA and with a significant number of fully saturated pixels.
To fill the frame with the same flat test chart one needs to shoot at a 1.6X greater distance with the crop body than with the full frame body. If you use 10 feet for the full frame body then you must shoot at 16 feet with the crop body. What you are not increasing, however, are the number of lines per inch (as projected onto the sensor) that the lens is capable of resolving. The image of the subject cast by the lens is smaller at 16 feet than at 10 feet, and so the lens' resolution limit is wider in relation to the surface features of the subject and to the size of each pixel (assuming the APS-C and FF sensor have the same number of pixels).
To get the same display size the image from a crop body must be enlarged by 1.6X more than with an image from a full frame body. For a 4x6 print the full frame image need only be magnified at about a factor of 4.23 compared to 6.77 for the crop body image.
With both the greater shooting distance (1.6X) and the greater magnification (1.6X) you are stretching the resolution limits of the lens to a greater degree (2.56X). To put it another way: to get the same sharpness with the crop body you would need a lens capable of resolving 1,800 lines per inch to equal the full frame camera with a lens capable of resolving 700 lines per inch!
Even if you have both an 80mm lens for the FF camera and a 50mm lens for the crop body so that you could shoot at the same distance, you would still need the 50mm lens used on the APS-C body to resolve about 1125 lines per inch to equal the 700 lines per inch 80mm lens used on the FF body, because you are still magnifying the result by 1.6X more to get the same display size.
For simplicity's sake regarding the math, the following theoretical illustration assumes an APS-C sensor that is 1.5X smaller than the FF sensor (even though the original question is in regard to a camera with a 1.6X crop factor sensor).
Imagine that you have a lens with theoretical resolution limit of 1000 line pairs per mm. With a 24mm wide sensor it could project 24,000 line pairs. With a 36mm wide sensor it could project 36,000 line pairs. Now take a test chart with 36,000 line pairs that fills the frame of the FF camera at ten feet. If you back up to 15 feet to fill the frame of the crop body camera with the same test chart then the 36,000 line pairs on the test chart are going to exceed the resolution capability of the lens because there are 36,000 line pairs trying to fit on a 24mm wide sensor.
You don't back up because the lens magnifies more when attached to a crop body. The lens projects the same size image either way. The reason you back up is to allow the smaller sensor to capture the same framing. This reduces the angular size of the subject by 1/1.5X in the virtual image actually projected by the lens. But you don't reduce the angular size of the resolution limit of the lens by 1/1.5X by backing up.
At 15 feet from the chart the angular difference between each line pair is 1/1.5X the angular size when the camera was 10 feet from the chart. But the lens still has the same resolution limit that is ultimately based on the angular size of the line pairs on the test chart. Line pairs per mm can only be meaningful when the distance from the lens' entrance pupil to the sensor remains constant as well as when the magnification factor from the virtual image projected on the sensor to a particular display size remains constant.
You then enlarge the APS-C image 1.5X more than the FF image in order to view both images at the same display size. This means that with the image from the APS-C sensor we can perceive blur circles (as measured on the sensor prior to display magnification) that are 1/1.5X the size of blur circles at the limit of our perception on the FF image. Slightly blurred edges that would look sharp in the FF image can be seen as blurry due to the greater magnification of the APS-C image.
If the 1.5x crop body image of a 24K line pair chart taken from 15' is printed at 4x6 and the FF image of a 36K line pair chart taken at 10' is printed at 6x9, then the sharpness should be the same because the line pairs would be the same width on both prints. But when you print the 1.5 crop body image at 6x9, the line pairs (which are at your lens' resolution limit) are now 1.5X wider. You don't gain any additional subject detail by enlarging more, because the lens can't resolve those details smaller than the width of the line pairs. At that point you are only revealing blur.
The two effects are multiplied: pulling back for the same framing reduces the angular size of subject details by 1.5X, then magnifying by 1.5X more to display at the same size decreases the acceptable Circle of Confusion by a factor of 1.5X.
Here's another way to look at it: If the 1.5x crop body image of a 24K line pair chart taken from 15' is printed at 4x6 and the FF image of a 36K line pair chart taken at 10' is printed at 6x9, then the line pairs would be the same width on both prints. Keep in mind that the FF image is resolving 36K line pairs displayed at 6 x 9 inches, while the 1.5X crop body is only resolving 24K line pairs displayed at 4 x 6 inches. But when you enlarge the 1.5 crop body image to 6x9, the line pairs (which are at your lens' resolution limit) are now 1.5X wider.