Camera Labs

Hi folks,

Edit: I was having a bad hair day when I composed the following post as detailed in my post below. I was tempted to ask myself, in my capacity as a Mod, to delete the thread but that would be cheating. I'll leave one of my more inglorious forum moments in place as a reminder to myself to be more careful in future!

Are high ISO noise tests fair when comparing cameras with different sensor sizes?

A typical testing methodology goes something like "setting the zooms to the same angle of view" and then using aperture priority to capture images at various ISO. Perfect if one was comparing different cameras with the same size of sensor but not so perfect otherwise.

Take an extreme example of two sensors with the same pixel count, one of which has a crop factor of 1.5 and the other a crop factor of 1 (i.e. full-frame) and, to keep the numbers simple, I'll assume the f-number used is f/4. Test the 1.5x crop camera with a lens zoomed to 48 mm and the entrance pupil of the light entering the lens has a diameter of 48/4 = 12 mm. The full frame camera must have its lens zoomed to 72 mm to get the same field of view so, as it is also being tested at f/4, the entrance pupil has a diameter of 72/4 = 18 mm. The full frame camera's lens is sucking in light at 1.5 x 1.5 = 2.25 times the rate of the cropped sensor camera and that means the rate at which the photons hit each pixel on the full frame sensor is also 2.25 times greater as I initially posited that the two sensors had the same pixel count.

OK, this is an extreme example but, if I may acknowledge and then ignore the added complication of different pixel counts and aspect ratios, it seems pretty common for reviewers everywhere to compare micro four-thirds and APS-C sensors in this way and this skews the result in favour of the APS-C cameras by a factor of about 1.8 times when one considers the entrance pupil of the lens.

To be fair the reviewers are between a rock and a hard place. If they actually tried to measure sensor performance by throttling the aperture (smaller f-number) of the larger sensor cameras they would get masses of complaints by those who just looked at the f-number used. And the camera manufacturers also make a rod for their own backs as I don't detect any trend for the manufacturers of the higher crop factor cameras to offer those cameras with brighter (lower f-number) kit lenses - if they did they could certainly cry "foul" with some justification if the "same f-number" approach to testing were used.

There are so many variables to juggle just to try to measure something so apparently simple as high ISO noise and to then try and draw meaningful conclusions from those measurements that it makes my head spin. Unless one wants to get into the minutiae of technical measurements, such as with DxO, maybe the current approach is best. Or do you disagree?

Bob.

_________________
OM-D E-M5 + ED 12-50mm, H-PS14042E, H-F007014E, L-RS014150E, M.ZUIKO DIGITAL 45mm 1:1.8.
Gitzo GT1541T, Arca-Swiss Z1 DP ball-head.
Astrophotography: TEC 140 'scope, FLI ML16803 camera, ASA DDM60 Pro mount.

It's a tough one, even as you suggest, in the case where the sensor is a similar size let alone a different one.

The question of depth of field equivalence is often raised as another method of comparison, but I think for most people they're happy to think what I call "exposure setting priority", even if in practice the output will be different. micro4/3 users are often keen to point out the "advantage" of the crop factor in delivering smaller lenses at longer effective focal lengths, putting aside the depth of field and related considerations.

Personally I think there is one thing that has been neglected by all mainstream noise comparisons, even DxO. That is the nature of the noise as well as quantity. At a crude level, most people prefer a fine grain over a coarse grain, and colour noise is more noticeable too. I think it is more complicated than that, for example it may also include different contributions from fixed pattern noise and hot pixels too. All very complicated so good luck to anyone coming up with a system that can handle it.

Until then I think we'll continue to have assorted narrower view comparisons which may or may not represent how anyone might really use it.

_________________
Canon DSLRs: 7D, 5D2, 1D, 600D, 450D full spectrum, 300D IR mod
Lenses: EF 35/2, 50/1.8, 85/1.8, 135/2+SF, 28-80 V, 70-300L, 100-400L, TS-E 24/3.5L, MP-E 65, EF-S 15-85 IS
3rd party: Zeiss 2/50 makro, Samyang 8mm fisheye, Sigma 150 macro, 120-300 f/2.8 OS, Celestron 1325/13
Tinies: Sony HX9V.

Hi Bob,

that is an interesting question!

If I look at it from a potential buyer's perspective, I don't really care what the root cause of the noise is - whether the noise comes from difference in sensor-size, difference in generation of the camera/sensor even if the sensors are same size, difference in processor or whatever the cause may be.

What I DO care about is purely: How much noise does this camera exhibit at different ISO levels.

From the point of view that I'm deciding if I want to buy a specific camera or not, I care about how well this camera - compared to other cameras not matter their price, sensor size etc. - handles noise. Not just the amount, but the nature of it, of course.

I'm not blind to the dynamics of comparisons in absolute terms and in relative terms. Most reviews take that into account in other dimensions anyway. Still, they all list FPS in absolute terms, but may comment that it's good, bad, average for this "class" - still, I would not want another FPS-like metric to articulate speed for a given class, because it forces me to understand the "class" to understand how a given camera performs - and that is an unnecessary extra learning curve that adds no real value, in my opinion.

...but that's just me

Cheers!

_________________
Nikon stuff and other stuff, but not nearly as much stuff as I really want!

I'm totally with you on that. And when the reviewers are testing compact and superzoom cameras the root cause of the noise is pretty irrelevant though I can certainly make a case that all such tests should be performed with the aperture at its brightest available setting for the field of view.

At the risk of becoming a little obsessive one or two forum members may have noticed that I'm ploughing a bit of a lonely furrow when it comes to the merits or otherwise of larger sensors. We regularly hear the mantras "smaller sensors mean more image noise" and "larger sensors offer shallower depth of field". To a first approximation, so far as I can see, neither statement is true. The primary factor is the size of the entrance pupil of the lens used for a given field of view and not the sensor size. Yes, it's easier to make lenses with larger entrance pupils for a given field of view when the sensor is larger so in practice that means that high crop factor (smaller sensor) interchangeable lens cameras tend to have smaller diameter lenses and so noisier images with greater depth of field but it is those lenses which are the root cause of the problem and not the sensor per se.

Smaller pixels may also be less efficient (circuitry on the silicon chip blocking photons) and might even have noisier electronics (not sure about that one) but I wish it were possible to conduct reviews when comparing cameras with different sensor sizes in a way which didn't handicap the smaller sensor cameras unnecessarily. I somehow don't think interchangeable lens camera reviewers are going to change their ways (not practical in the real world) but in my perfect virtual world I'd like to see:

1. Side by side photographs from different cameras being shot at the brightest available aperture of the standard kit lens when doing the high ISO comparisons. That tells a buyer how good the images can be without investing in more glass.
   
2. Testing of the sensor's inherent noise and dynamic range. Unfortunately because of the granularity in setting both f-number and shutter speed, I don't see how one could compensate for sensor size short of having an infinitely variable lighting setup or a full blown test bench.

Phew, I am so glad I don't review cameras for a living!

On reflection, that's probably a view shared by most who've just read what I've written above!!

Bob.

_________________
OM-D E-M5 + ED 12-50mm, H-PS14042E, H-F007014E, L-RS014150E, M.ZUIKO DIGITAL 45mm 1:1.8.
Gitzo GT1541T, Arca-Swiss Z1 DP ball-head.
Astrophotography: TEC 140 'scope, FLI ML16803 camera, ASA DDM60 Pro mount.

On Bob's point 1, you're then reviewing the package, not just the camera or the lens by itself. In a parallel sense while generalisations are often made about sensors, they do not usefully function without a lens so that is only implicitly a consideration.

On point 2, while the settings are indeed stepped, I think we have to look at significance here. We can only set typically to 1/3 EV steps, giving a maximum error of matching of two devices to 1/6 EV. I think for most people that is insignificant at any practical level.

_________________
Canon DSLRs: 7D, 5D2, 1D, 600D, 450D full spectrum, 300D IR mod
Lenses: EF 35/2, 50/1.8, 85/1.8, 135/2+SF, 28-80 V, 70-300L, 100-400L, TS-E 24/3.5L, MP-E 65, EF-S 15-85 IS
3rd party: Zeiss 2/50 makro, Samyang 8mm fisheye, Sigma 150 macro, 120-300 f/2.8 OS, Celestron 1325/13
Tinies: Sony HX9V.

Hi folks,

On reflection I think I need to make an apology as my posts in this thread haven't been up to the standard I usually try and reach. I'll try and put that right.

The elephant in the roomy space that used to hold my brain is what exactly it means when one sets a particular ISO. For a given level of illumination and a correctly exposed image:

Exposure time x ISO / f-number = a constant

So, for a given ISO during high ISO noise tests it doesn't really matter what the f-number (or entrance pupil) is so long as the final image is correctly exposed by varying the exposure time.

Of course there's another elephant waiting out in the wings ready to take centre stage and that is what ISO actually is with reference to digital sensors. I started a topic on that some years ago (Know your Base (or Native) ISO).

I won't rehearse that thread again here except to observe that smaller full-well capacities (as typically associated with small pixels on small sensors) mean less electrons for a fully exposed pixel. That may well be sufficient to produce clean images at low ISO numbers but when the ISO is ramped up by increasing the gain of the read out amplifiers (and the resultant image is produce by fewer photons releasing electrons) there are so few electrons that the resultant gradations in luminance and colour become a lot less than smooth in the darker areas of the image. The image becomes noisy.

Again, my apologies for running with a bad argument at the start of the thread. I'll stick with my views about sensor size and depth of field but given the close link between pixel size and full-well capacity I was flat wrong elsewhere in the thread.

Bob.

_________________
OM-D E-M5 + ED 12-50mm, H-PS14042E, H-F007014E, L-RS014150E, M.ZUIKO DIGITAL 45mm 1:1.8.
Gitzo GT1541T, Arca-Swiss Z1 DP ball-head.
Astrophotography: TEC 140 'scope, FLI ML16803 camera, ASA DDM60 Pro mount.

As a sidenote, when testing the noise levels, it's also important to check the actual exposure every time you double the ISO. Sometimes some cameras cheat and their 1600 ISO is actually more like 1000 or 1200 ISO. Some are also simply less sensitive at xx ISO than another camera at the same ISO. I always try and mention this if I spot anything fishy!

PS - shouldn't this be in the off-topic section?

Hi Bob,

I do see what you mean in terms of the test approach. Some sort of baseline approach would be good when it comes to comparisons.

The depth of your musings reminds me of discussions that were held..what..about 20_ years ago when it comes to hi-fi. I'm thinking specifically about testing amplifiers and the different schools of thought...very technical measures vs. listening tests.

Technically the tube amps scored very poorly in all tests, but they scored very high in listening tests.

Sometime when I've read too much technical details about any one field, I find myself simply confused at a higher level than before and feel like I've just asked a silly question out loud: "Which brushes - wide or narrow - produce the best paintings?".

With VR/IS and overall low-light performance improving, I pragmatically find myself setting the limit at ISO800 for my D300 and ISO3200 for my Fuji X100 because I've simply found that on 18X12inch prints, that's my personal threshold.

Sorry, I don't mean to direct your topic onto a tangent - I know you're examining that holy grail of bias free testing and I think my profession has damaged me to this notion..lol.

Cheers
Ken

_________________
Nikon stuff and other stuff, but not nearly as much stuff as I really want!

Hi Ken,

Just for a moment there I thought Gordon had been promoting new members to the rank of Moderator!

@Gordon - I initially put the thread in "Rules, feedback and new member intros" under the "feedback" heading but I was in two minds. Actually, given the quality of the post I was probably in more than two minds, several of which I disown. I'll move the topic across to "Off-topic".

Bob.

_________________
OM-D E-M5 + ED 12-50mm, H-PS14042E, H-F007014E, L-RS014150E, M.ZUIKO DIGITAL 45mm 1:1.8.
Gitzo GT1541T, Arca-Swiss Z1 DP ball-head.
Astrophotography: TEC 140 'scope, FLI ML16803 camera, ASA DDM60 Pro mount.