Musings on Photography

Making Comparisons

Posted in equipment, process, technique by Paul Butzi on January 13, 2009


Over on The Online Photographer, Ctein writes compellingly about the perils of judging cameras by numeric specifications

A distressing number of comments that I read run along the lines of “I would never buy camera X because…it’s made by A and B makes better cameras/it has a small sensor and they’re all crap/it has too many pixels to produce good quality/etc.” Comments like that are not the hallmark of savvy shoppers. Broad rules of thumb are good ones to tell your neighbor or relative who doesn’t know much about cameras, when you don’t want to take the time to look at individual models for them. Very sensible. But when it comes to your own shopping, it’s not.

Ctein’s point, I think, is both well put and important. Often, there are ongoing arguments about differences which show in the numbers but which don’t show in the images. Beyond his point about significance of the numbers, there’s another peril lurking behind those numbers – the methodology. Simply put, camera manufacturers might not lie outright, but they will certainly adjust their methodology to get the result most favorable to them. And because each manufacturer has a different methodology, you can’t compare the numbers from manufacturer A and the numbers from manufacturer B without being forced to compare apples to something much more different from an apple than an orange would be. Apples to aardvarks, say.

Beyond the perils of manufacturer’s statistics, though, you even run into problems if you’re performing the comparison directly, all by your little own self. You think that you’re comparing the dynamic range of your Sony Whoozywhatsit to the dynamic range of your Canon Whatchamacallit, and you don’t get very far before you run into problems with experimental design.

The big problem is this: it’s not particularly hard, especially with digital cameras, to measure something fairly accurately and with fair precision. (side note: if you’re not aware of the difference between accuracy and precision, you’re lost before you begin). The problem begins, as it does with all experiment, in measuring just one thing. That is, it’s pretty easy to measure the dynamic range of your camera, combined with the linearity of your monitor, the sensitivity of your eyes, the temperature of the room, how long your monitor has been turned on, and a host of other lesser variables. What’s hard is excluding everything except dynamic range from your measurement.

And it’s not just dynamic range. The same problem occurs when you try to measure anything about your process at all. This was particularly a problem with darkroom lore, where badly designed experiments had proven many, many things that just weren’t so. Way back when I wrote an article for Photo Techniques that pointed out that cyan filtration couldn’t have any effect on variable contrast papers, I got a raft of letters claiming no end of magical properties for introducing cyan filtration. Because I’m particularly hard headed, I wrote back to each and every one of those correspondents, suggesting that they should try the following: go into the darkroom, and make one straight print, process it, and then make another print using the identical exposure time, filtration, etc., and process the second print. The two prints should be identical at least to the ability of the human eye to detect. If they weren’t, then there are uncontrolled variables that need to be excluded before any experiment regarding cyan filtration can begin. Some number of people wrote back, saying that this was an eye-opening experience – they’d gone into their darkroom expecting it was child’s play to produce two identical prints back to back. It turns out that most darkroom workers don’t time print development, don’t have consistent print agitation, don’t properly test for safelight safety, don’t control the developer temperature, and often have unreliable enlarger timers and outright white light leaks. Or, to put it another way, most darkrooms are supped full with uncontrolled variables, and thus are horrid places to conduct experiments. So people would conduct experiments, and any valid experimental result was swamped by the effect of the uncontrolled variables, and the results were just so much superstition. Although it sounds easy to make two identical prints back to back, it’s actually anything but.

Lots of people seem to think that moving into the digital age has eliminated all this stuff – computers are nothing if not repeatable, right? But it turns out this just isn’t so – we’re still plagued by the same sorts of uncontrolled variables when we go to compare prints.

Here’s a little test you can make – a variation on the ‘identical print’ test. Make two prints of a single image. Same size, same paper, from the same batch of paper and off the same printer, back to back. These prints should be identical, right? Right.

Label the prints, then take the two prints, a pad of paper, and a pen, and go to your work table (or desk, or kitchen table). Sit down, place the two prints side by side, and start looking for differences. When you find a difference, write it down. Don’t worry, if you look carefully enough you will find differences, trust me. Put in some ten or fifteen minutes, and really examine everything. Look at tonal smoothness. Look at shadow detail. Look at highlight detail. Look at overall tonal distribution. Check for differences in overall color, and for color differences in the different corners of the print. Every time you find a difference, no matter how minor, write it down.

After fifteen minutes, swap the position of the prints – that is, take print A and put it where print B was, and put print B where print A was. Run through your list of differences. How many of them went with the print, and how many of them stayed with the position? It’s a safe bet that a fair number of the differences stayed with the position, because the probability of the lighting on both positions being the same is zero, unless you happened to have picked a place to view the prints where you’ve gone to some significant trouble to ensure that both spots get exactly the same lighting. The light from windows will unevenly illuminate them, if nothing else, so that the position that is closest to the window will be yellower or bluer, depending on the weather. If a nearby wall is strongly colored, then one position will tend toward that color more than the other position. One position will be slightly dimmer.

This is an interesting test to run, if only because it makes it clear just how much our process we’ve left uncontrolled. We can sit at the computer, endlessly making very small tweaks to an image, making evaluation print after evaluation print, and comparing them side by side on our work tables, only to have the small changes we make get swamped by the changes in light on our work table that occur on a partly cloudy day.

6 Responses

Subscribe to comments with RSS.

  1. Philip said, on January 13, 2009 at 9:25 am

    This is a great post. The scientific materialism of the currently dominant worldview has taught us to do comparisons between consumer products to determine which one is “better,” but controlling all the variables (including sample-to-sample variation) is beyond the resources and “ganas” of most individuals.

    Surely some sort of 80-20 rule applies when it comes to controlling enough of the process to produce consistent work, and it would be interesting to hear some commentary on that subject!

  2. Erik said, on January 13, 2009 at 11:50 am

    I’ve got a friend who’s been talking to me about horrible noise on pictures from his D70 (I’m a Canon freak…). He tried all different ISOs, and different processing software. Finally, he got me to look at some of them and I didn’t see any noise. Then he noticed he couldn’t see it on his laptop.

    His monitor has such an elevated black point that it brings out all the noise in the shadows.

    He calibrated it, but either the calibration or the monitor is defective. He thought he just had a crappy sensor and my Canon bigot nature didn’t help with that.

  3. Bryan Willman said, on January 13, 2009 at 8:09 pm

    Clothing colors. Do you always wear an all white or all black shirt? No. Guess what, your body covered by a garment with strong colors is a real reflector and will change the balance of the image.

  4. Gordon McGregor said, on January 14, 2009 at 7:37 am

    Bryan’s point is a good one – I’ve used that to my advantage quite often for street portraiture. You can become your very own fill card, particularly good on sunny days – stand in the sun, have the person in open shade, be wearing a white top…

  5. Gordon McGregor said, on January 14, 2009 at 7:38 am

    forgot to add, almost my entire wardrobe is in neutral tones/ grey hues from white to black. I think that’s accidental, but it is true.

  6. Mike Harris said, on January 15, 2009 at 9:52 pm

    Wonderful post, I would much rather read this than another “my equipment better than yours”. This seems to fall under the category of whatever you say is really about you. It would follow that whatever you do is about you or greatly influenced by you. Until you can be removed from the experiment you will flavor that experiment. This is starting to sound like the Uncertainty Principle.

    And who’s to say your eyes see blue exactly the way my eyes see blue?

Comments are closed.

%d bloggers like this: