Camera Labs

Many of you know that I'm reviewing lenses in my spare time (see e.g. my post in the Nikon, Sigma and Tamron section). I'm trying to give you an overall impression mostly of sharpness (and contrast, as both factors are deeply intertwined).
Now here's the 10,000$ question: How do you measure sharpness which depends heavily on quality of focus?
This in turn splits into several sub-questions:
- Should I use AF or MF?
- Should I repeat AF coming from different starting distances? Like if the subject is 2m away should I defocus the lens first to 1m and/or 5m and then let it find it's way?
- If using MF how can I be sure that I focussed correctly?
- Should I "bracket" focussing? How do you bracket? Certainly not by moving the focus-ring

Some people must already have answered all these questions! Yes they have: In a professional lab, the resolution-test-charts are shot with the camera (+lens) moved in small increments relative to the test-chart to make sure that you nail the focus at some position.
Now, what happens if the lens has field-curvature (that is, the plane of sharpest focus is not a plane but some curved area)? There are two principle possibilities:
(1) The lab analyses only the shot that is sharpest at its centre and measures the resolution of this single shot at the borders and corners. If the lens produces a flat field of sharpest focus this is ok. If the lens has field-curvature the borders and corners show a weaker resolution in this single shot than at some different distance.
(2) The lab takes the best (=sharpest) shot for each section of the image. In this case the sharpest corners may come from a different shot (=shooting distance) than the sharpest center. If the lens produces a flat field of sharpest focus the selected shot(s) will be the same as in (1). If the lens has field-curvature, the lab will measure better results at the borders and in the corners than with method #1 because it analyses different shots.

Now the question is: Do you shoot flat subjects or is it not really relevant for you whether the lens has field-curvature or not?
Well, I cannot answer this for you, but I can tell you, what answers you get from my reviews:
- I always analyse a single shot of a flat subject
- But I select the best AF-shot from a bunch
- I always use AF.
- If the results from the different corners yield different resolutions I use the corner/border with the best results.

The consequences are:
- If the AF is consistently off, my results will not show the potential of the lens even if the lens could produce fantastic results with MF. My take on this: (a) nobody can focus manually as good as a good AF and (b) most people use AF so this is the most likely scenario!
- If the AF is just like a scatter-shot I take the best result for analysis. If you own such a lens, make sure to try multiple AF-efforts to have a good chance to nail the shot.
- If the lens is decentered (which most lenses are more or less) taking the best corner gives you at least an approximation of what this lens(-design) is capable of if it only were well centred.
- If the lens has field curvature, my review shows you, what to expect when shooting flat subjects. But I will/can not indicate whether a lens has field curvature. So it just might be that some of the lenses where I complained about weak borders/corners are fantastic in the field with non-flat obbjects

So that's about it and how I decided to cope with these problems. You now know about it - and you can never say I didn't tell you

If you'd ask me what I think many test-labs use I'd say: mostly #2 but with Gordon's tests of corner sharpness I think he uses the image that gives the best centre-sharpness (= method #1). But we can ask him!
Gordon? GOOORDON???

_________________
Thomas (beware: Nikon-fanboy and moderator!) My Lens Reviews, My Pictures, My Photography Blog
D800+assorted lenses

Thanks for sharing Thomas. So...when lenses are tested the corner resolution and centre resolution figures are taken from the shots which give the best results, irrespective of the differences between the two shots? What if the centre of the frame reaches it's highest resolution at f/4 and the corners at f/11?

_________________
Graham
Flickr

Hi Thomas, if you're referring to the ISO chart, I always focus on the middle whether I'm subsequently going to crop from the middle or corners.

I always start with AF, but sometimes need to tweak this. If the camera has sensor-based Live View, I use the magnified assistance option to check the focus. Otherwise, it's a case of retaking shots with adjusted focus if necessary after analysing them on the computer. The same applies to my outdoor resolution and indoor ISO tests - in fact if anyone is doing a tripod-based shot and has live view, I'd urge them to see if they can in fact get a sharper result by manually tweaking the focus (in magnified assist mode) rather than always rely on the AF to get it right. You may be surprised.

Graham, my test shots are always optimised for the centre.

Gordon

@Graham:
If you look at the results from e.g. photozone you can see separate curves for center- and border-resolution. And yes there are lenses, where the center peaks e.g. at f/5.6 and the border-res at f/8.0.
Using such a lens in the field (i.e. shooting real photos and not test-charts) can give you visibly different distributions of sharpness across the frame depending on the aperture you use.
In my reviews I always say at which aperture a shot (or crop) was taken.

@Gordon: Thanks for sharing. Yes, live view is a great tool to get the best focus! And I'm not really sure whom to blame when AF doesn't work optimally: the lens or the camera's AF system

I will have a look at the newly reported results from best/worst AF in the latest lens-reviews from ColorFoto (the German mag). Maybe a pattern occurs.

_________________
Thomas (beware: Nikon-fanboy and moderator!) My Lens Reviews, My Pictures, My Photography Blog
D800+assorted lenses

Thanks for the replies Gordon and Thomas.

The f/4 and f/11 was an exaggeration btw.

_________________
Graham
Flickr

Here's some overview of ColorFoto's lab-test on AF quality. To measure this they shoot 10 shots with AF and give the achieved contrast relative to a best manual focus result (as the benchmark). So these results can never be better than 100% and if you see a best AF value of say 80% that means that even taking 10 AF-shots the best shot of the bunch just reached 80% contrast of the MF results
Now, what I looked first into was the 70-200mm class of lenses as the problem of inconsistent AF was haunting me with my Tamron 70-200/2.8 review. Here are the results at different focal lengths. The 1st number is the contrast of the worst AF-try, the second number the contrast of the best AF-try out of 10 shots, all measured relative to the best you can get with MF. I've marked all results below 60%.
- Nikon 70-200/2.8 on a D300: @70mm: 82-99%, @120mm: 94-100%, @200mm: 85-99%
- Tamron 70-200/2.8 on a 450D: @70mm: 25-98%, @120mm: 33-83%, @200mm: 44-99%
- Canon 70-200/2.8 on a 450D: @70mm: 83-100%, @120mm: 24-81%, @200mm: 22-97%
- Sigma 70-200/2.8 on a 450D: @70mm: 27-90%, @120mm: 40-83%, @200mm: 49-96%
- Sigma 70-200/2.8 on a D300: @70mm: 65-77%, @120mm: 99-100%, @200mm: 58-94%
Well, this confirms my thoughts: AF quality is swinging wildly. Even if you (safely) assume, that 20% difference in contrast is not easily visible there are many figures below the 80% mark above.
Interesting is also the comparison of the Sigma on Canon vs. Nikon: The difference in performance might be the result of sample variations of the Sigma lens not different AF-quality of the bodies.

Looking at another lens I've already reviewed, the Sigma 120-400mm, the following results came up in the lab-test:
- Sigma 120-400 on a D300: @120mm: 96-100%, @220mm: 80-91%, @400mm: 76-80%
- Sigma 120-400 on a 450D: @120mm: 64-90%, @220mm: 69-98%, @400mm: 77-93%
Looking at the results on the D300 at 400mm (max 80%!) that might explain some of the results I got.

Interesting results also with the Sigma 150-500mm:
- Sigma 150-500 on a D300: @150mm: 92-93%, @270mm: 99-100%, @500mm: 100-100%
- Sigma 150-500 on a 450D: @150mm: 40-81%, @270mm: 30-70%, @500mm: 38-54%
This is a huge difference in AF-quality on the Canon vs. the Nikon. Again: might be sample variations of the lenses. Looking at the resolution charts the Sigma copy for Canon looked worse at 270mm than the copy for Nikon mount, which is also confrimed by a higher decentering score at 270mm. But at 150mm and 500mm both copies seemed to work at comparable quality.

_________________
Thomas (beware: Nikon-fanboy and moderator!) My Lens Reviews, My Pictures, My Photography Blog
D800+assorted lenses

Interesting results, Thomas. Your data suggests to me that the Canon AF is vastly inferior to the Nikon AF.

Or you somehow managed to end up with terrible lens samples on the Canon.

_________________
PhilipGoh.com

I've been so frustated by sample variations of lenses so far (look e.g. at my Sigma 300/4.0 review with 3 lenses tested) that I would never exclude this as the culprit!
But when I look at the IQ results for the tested lenses you might infer that the tests also show some weakness of the 450D AF compared to the D300 AF. But you can clearly state that the bane of inconsistent AF can also ruin a lens on a body with reportedly very good AF

And for the record: the values in my last post are from the ColorFoto lab test, not my own efforts. As I said: I normally use AF only to review the quality of a lens.

_________________
Thomas (beware: Nikon-fanboy and moderator!) My Lens Reviews, My Pictures, My Photography Blog
D800+assorted lenses

Is that a known issue with Sigma lenses? I know that many people say that Sigma's QA is really shoddy and you can end up with a lemon. There are reports of people receiving lemons 3x in a row after sending away a lens that was defective only to have it replaced a couple of times with one that was equally defective.

Then again, I don't buy many lenses so all I got is hearsay.

_________________
PhilipGoh.com

I don't think any system is immune to this - I've experienced small focus errors with most systems, and you may not notice until you compare them against a shot that's been taken with manual focus that's 100% (or close to being) spot on. And this is again an area where I find Live View - or at least those Live View systems with a proper 100% magnified view - absolutely invaluable.

Don't get me wrong, the difference can often be very subtle, but there is normally a benefit to ensuring the focus is spot-on rather than hoping the depth of field will cover it.

You can end up with a lemon from almost any manufacturer. This makes it absolutely crucial to test your lens asap and send it back if it does not satisfy you.
I've seen many decentered Nikon lenses in the tests too, e.g. the Nikkor 70-300 VR was decentred 27% at 145mm, the Nikkor 80-400 was decentered 32% at 80mm etc. etc. But I think that the focus reliability may depend on something else: I had some strange occurences e.g. with a Nikkor 50/1.4 and also with the Tamron 70-200/2.8 at 200mm where I suspect that a particular aberration (perhaps coma) makes it hard for the phase change AF systems to detect the best focus. As I wrote in my review even the live view contrast based AF had no chance to focus the Tamron properly. Btw. another site reviewed the same lens and remarked "not a systematic 'front-focus' or 'back-focus' problem, but instead an apparently random tendency to miss focus slightly" on any Canon body they tested. So this certainly does not look like "sample variations" but a problem that is "designed" into the lens.

_________________
Thomas (beware: Nikon-fanboy and moderator!) My Lens Reviews, My Pictures, My Photography Blog
D800+assorted lenses

is it possible that temperature and humidity effect the sharpness and focusing?

_________________
Jake O'Connell, 40D Crew
Canon EOS 40D | 28-135mm IS | 50mm f/1.8 |Vivitar DF 383 | Vivitar 285hv
My Flickr
my Blog
"Photography isn't so much about the results as it is the collective experience, your interactions with people and with the world"

Maybe, Jake. But you can bet that the lab-test of ColorFoto and my own test were done under almost ideal climatical conditions. So temperature and humidity are not the excuse for poor AF reliability.

_________________
Thomas (beware: Nikon-fanboy and moderator!) My Lens Reviews, My Pictures, My Photography Blog
D800+assorted lenses