Edit: I was having a bad hair day when I composed the following post as detailed in my post below. I was tempted to ask myself, in my capacity as a Mod, to delete the thread but that would be cheating. I'll leave one of my more inglorious forum moments in place as a reminder to myself to be more careful in future!
Are high ISO noise tests fair when comparing cameras with different sensor sizes?
A typical testing methodology goes something like "setting the zooms to the same angle of view" and then using aperture priority to capture images at various ISO. Perfect if one was comparing different cameras with the same size of sensor but not so perfect otherwise.
Take an extreme example of two sensors with the same pixel count, one of which has a crop factor of 1.5 and the other a crop factor of 1 (i.e. full-frame) and, to keep the numbers simple, I'll assume the f-number used is f/4. Test the 1.5x crop camera with a lens zoomed to 48 mm and the entrance pupil of the light entering the lens has a diameter of 48/4 = 12 mm. The full frame camera must have its lens zoomed to 72 mm to get the same field of view so, as it is also being tested at f/4, the entrance pupil has a diameter of 72/4 = 18 mm. The full frame camera's lens is sucking in light at 1.5 x 1.5 = 2.25 times the rate of the cropped sensor camera and that means the rate at which the photons hit each pixel on the full frame sensor is also 2.25 times greater as I initially posited that the two sensors had the same pixel count.
OK, this is an extreme example but, if I may acknowledge and then ignore the added complication of different pixel counts and aspect ratios, it seems pretty common for reviewers everywhere to compare micro four-thirds and APS-C sensors in this way and this skews the result in favour of the APS-C cameras by a factor of about 1.8 times when one considers the entrance pupil of the lens.
To be fair the reviewers are between a rock and a hard place. If they actually tried to measure sensor performance by throttling the aperture (smaller f-number) of the larger sensor cameras they would get masses of complaints by those who just looked at the f-number used. And the camera manufacturers also make a rod for their own backs as I don't detect any trend for the manufacturers of the higher crop factor cameras to offer those cameras with brighter (lower f-number) kit lenses - if they did they could certainly cry "foul" with some justification if the "same f-number" approach to testing were used.
There are so many variables to juggle just to try to measure something so apparently simple as high ISO noise and to then try and draw meaningful conclusions from those measurements that it makes my head spin. Unless one wants to get into the minutiae of technical measurements, such as with DxO, maybe the current approach is best. Or do you disagree?