DCRAWIntriduce

WHAT IS DCRAW
and endless list of RAW formats expanding continuously. In words from the author: "So here is my mission: write and maintain an ANSI C program that decodes any raw image from any digital camera on any computer running any operating system". And so far he has achieved that.
DCRAW works in Linux so as in Windows and Mac. In this tutorial we will make use of the Windows version although it will work exactly the same way in the other two platforms since DCRAW has no graphic user interface being a 100% command line application.
DCRAW requires no installation, just copy the executable file into the appropiate pathname and invoque it from a terminal console.
There are several front-end interfaces that make DCRAW more easy to use such
as UFRAW, although in my opinion they lack of that extra of power and flexibility that DCRAW provides for not presenting to the user all its options in a straightforward way. In addition to this it is usual that a new feature or improvement appears every month, so any front-end gets quickly old fashioned. Mi advice: to learn how to use straight DCRAW.
DCRAW alone is not a friendly tool and because of this could not be ideal for your usual workflow. However thanks to the transparency and power of its low level control, it becomes the perfect tool to perform tricky RAW developments in specific cases such as:
▪Technical analyses and studies
▪Specially tricky to develop photographs
▪Comparision between RAW developers
My personal opinion is that I love DCRAW as a RAW development tool not only for experimenting but in a regular workflow. The key is to do in Photoshop all those extra features that other RAW developers can do apart from pure development, and consider DCRAW as a tool that provides a rough image without any kind of processing applied on it, just a high quality development under absolute control.
It is the only RAW developer I have found up to now that transmits the certainty of being performing on my RAW files exactly what I want to be applied to them and not what the RAW developer wants to; in fact as we will see it can do things that are out of reach in other more popular commercial RAW processors such as Adobe Camera Raw.
You will learn a lot about RAW theory and sensor linearity when using DCRAW. It is a piece of software that makes you feel very close to your RAW data.
DCRAW DOWNLOAD AND INSTALLATION
There are several online resources that keep updated DCRAW builds ready for download:
▪Manuel Lloren's site, RAWness. Digital Photography Science is the most recommended download site for Windows users. There we can find compiled versions for all Windows platforms (XP, Vista, Windows 7) both for 32 and 64 bits and with different levels of optimisation.
▪The Francisco J. Montilla site where Windows, Linux and Mac builds are offered. However the Vista and Windows 7 version are usually not updated to the last available DCRAW version since they require to be compiled using a Microsoft compiler.
▪The Benjamin Lebsanft site, with optimised versions for several platforms. It seems Benjamin is lately experiencing some trouble with WordPress' contents policy though.
Windows DCRAW versions require no installation since they consist of just a single compact .exe console application file. This file is named dcraw.exe and can be directly copied in the C:\WINDOWS\ path to be accesible from any folder when it is called in the command line.
HOW TO RUN DCRAW
DCRAW is executed from the command line in a MS-DOS window. To open an MS-DOS window go to Start → Execute and type cmd. A window with a black background will pop up and we will type dcraw on it so that DCRAW will display all available options:
Fig. 1 Command line displaying all DCRAW v8.82's available options.
In a real example of execution of DCRAW over an hypothetic RAW file named photo.cr2, we would open the former command line window, we would move to the folder containing the file (the CD command allows to move through our hard disk's folders) and we would type something like:
C:\>dcraw -v -w -H 1 -o 0 -q 3 -4 -T photo.cr2
DCRAW would display the following messages:
Loading Canon EOS 350D DIGITAL image from photo.cr2 ...
Scaling with black 256, multipliers 0.586287 0.421032 1.000000 0.421032
AHD interpolation...
Building histograms...
Writing data to foto.tiff ...
C:\>
Storing the result of the RAW development in the file photo.tiff.
DCRAW OPTIONS
writing single letters preceded by a dash symbol after the dcraw command, and are:
-v
Provides textual information about the RAW development process (recommended).
-e
Extracts the JPEG embedded in the RAW file, i.e. the JPEG file the camera generated for the camera display preview which differs from the JPEG file obtained when the camera is set in JPEG mode. It is a very quick way to get a JPEG view for all the RAW files contained in a folder. For instance doing: dcraw -e *.cr2.
-w
If DCRAW manages to find it, it will use the white balance that was adjusted in the camera at shooting.
-a
Performs an automatic white balance calculation over the whole image.
-r m1 m2 m3 m4
It sets a custom user white balance. These 4 values are the multipliers that w ill scale linear ly all levels found in the RGBG
channels in that order. The white balance means a scaling of image levels therefore all levels will move from their or iginal positions which could not be desirable in certain cases. If we are not to perform any white balance we will use the option -r 1 1 1 1. We will study this feature more deeply later as it is a very important concept.
-H [0-9]
With this option we w ill set the highlight behaviour being allowed the follow ing values: 0=clip, 1=no clip, 2=neutr al grey blown areas, 3-9=highlight recovery. This feature w ill be also studied deeply in the w hite balance and highlights section. I mainly use the -H 0 option for its linear ity and -H 2 when there is risk of blowing important parts of the image w ith the previous value. The -H 1 option guarantees that we w ill not blow any pr eviously non blow n channel but can lead to strange tones in the blown areas. The highlight recovery options are more sofisticated and slow down noticeably the execution speed.
-k n1 and -S n2
Respectively set the black and saturacion levels that w ill be used in the RAW development. Although it is best to let DCRAW calculate the black level, it is very interesting however to set the saturation level by ourselves for our particular camera w ith-S.
-D
Extracts an image w ith the pure RAW data without any demosaicing or scaling applied. It is very useful to analyse the real captured levels in the sensor's native r ange of 12, 14 or 16 bits.
-d
As the previous command, it does not perfor m any demosaicing but goes one step ahead in the development process since it adjusts black and saturation points, white balance and rescales to the output 16-bit range. It is very interesting to get linearized (for substracting the black point) undemosaiced data in a 16-bit scale w ith dcraw -d -r 1 1 1 1 that allows to obtain histograms in stops of exposure with Hi st og ramma r. It can also be used to study the Bayer pattern of the RAW file, permitting for instance to detect malfunction in the camera sensor or individual anomalous pixels:
Fig. 2 Hot pixel detection over Bayer pattern (move mouse over).
-o [0-5]
Sets the output colour profile being possible the values: 0=none (no colour management), 1=sRGB, 2=AdobeRGB,
3=WideGamut, 4=ProPhoto, 5=XYZ. To convert to a colour space means a matr ix transformation of the le vels of the image and in some cases this could not be desirably. Not to perfor m any transformation we will use the option-o 0.
-q [0-3]
Sets the quality of the Bayer demosaicing algor ithm employed. The more quality the more complex the algor ithm w ill be and thus less quick, however DCRAW is very fast in all of them. Possible values are: 0=bilinear, 1=VNG, 2=PPG, 3=AHD. I always use the last value w hich is an adaptive algorithm providing very good r esults, although according to the author DCRAW will use by default the best algorithm for each camera model. In this way for Fuji cameras the method-q 2 is better than -q 3.
Fig. 3 Crops at 200% of different interpolation quality values.
-4
It gener ates a linear 16-bit file instead of an 8-bit gamma corrected file which is the default. I always use this option.
-T
It outputs a TIFF image file instead of PPM.
-g gamma slope
Applies a gamma correction to the output defined by the gamma value and the toe slope of the curve. For a pure gamma curve set slope to 0. Some typical values are:
-g 1 1 linear 1.0 gamma (default if -4 is used)
-g 2.2 0 pure 2.2 gamma (Adobe RGB)
-g 1.8 0 pure 1.8 gamma (ProPhoto RGB)
-g 2.4 12.9 sRGB gamma
-g 2.222 4.5 BT.709 specification gamma (default if -4 is not used)
With the descriptions seen so far w e can now understand each of the parameters used in the example above C:\>dcraw -v -w -H 1 -o 0 -q 3 -4 -T photo.cr2:
-v Development show ing the progress infor mation on screen
-w We use the w hite balance that was configured in the camera
-H 1 We use a linear mode with no clipping in the highlights
-o 0 We do not convert the r esulting image to any colour space
-q 3 We set the maximum possible quality of interpolation
-4 -T We force 16-bit TIFF linear output
LINEAR HISTOGRAM
If we make use of the -4 option the development will produce a 16-bit linear output. This is the mode to w hich I w ill refer in the rest of the tutorial since it is the mode that allows to take advantage of DCRAW's main strongholds. A linear development means that no gamma correction has been applied yet (typ. gamma=2.2), and thus levels are distributed along the histogram in a linear way being each of them proportional to the amount of light (number of photons) that the pixel was able to capture during the exposure.
A linear image cannot be displayed due to its extreme darkness a s the histogram is densely concentrated on the left side of the range, but as we will se later PS allows to display correctly this kind of images without altering its internal data.
To see what a linear histogr am looks like let us take the follow ing scene:
Fig. 4 RAW development sample image.
The pr evious photograph developed w ith a gamma correction would have the following histogram distribution:
Fig. 5 Resulting histogram of gamma corrected development.
However when we develop it using DCRAW and the -4 option we get the follow ing linear histogram:
Fig. 6 Resulting histogram of linear development using DCRAW.
There are obvious differences. We can see how the information is concentrated in the lowest f-stops corresponding the whole second half of this histogram to only the last f-stop (marked as 12) which is almost empty as it only contains the highlights of the scene. The next quarter of the histogram corresponds to the next f-stop (marked as 11), and so forth up to completing the 12 f-stops we can achieve to capture using the 12-bit RAW sample file used.
We must not think that the higher concentration of infor mation due to the linear development means we are losing tonal richness. This is not like that at all, in fact a gamma corrected histogram starts from a linear one to w hich a gamma correction curve has been applied and that is why a lot of holes are created in the low end of gamma corrected histograms. The tonal richness is not incr eased nor reduced for using one or the other kind of image.
For being the image information in a linear for mat, any possible linear tr ansformation is very obvious; so the application of a white balance or exposure control become as we will see later very simple linear scaling operations over the levels.
WHITE BALANCE AND HIGHLIGHTS
As we commented befor e the commands to perfor m the w hite balance operation and choose the way we will process the highlights are respectively -w -a -r and -H. We w ill comment them in the same section for being closely interrelated. Depending on the highlights behaviour chosen, the w hite balance can lead to blow certain areas of the image that w ere not blow n in the original RAW file data. This is common to all RAW developers and we w ill study this in detail later.
WHI TE BALANCE
The way DCRAW implements the white balance interface is not the usual Temperature/Tint but a lower level set of 4 multiplying factors which will scale linearly each of the RGBG image channels of the Bayer matrix (white balance is applied before demosaicing). In general factors number 2 and 4 w ill be the same as they correspond both to the green channel, just in different locations of the Bayer distribution.
We must point that a particular white balance does not mean an absolute set of values for all those multipliers, but the relative proportions between them. In this way the same white balance can be achieved w ith different sets of factors as long as they keep their relative proportions.
The multipliers may have 3 different or igins depending on the option used:
-w White balance is set according to camera settings at the moment of the shot
-a Automatic white balance calculated by DCRAW over the whole image
-r m1 m2 m3 m4 Custom w hite balance by chosing the individual multiplying factors
If none of these options is used, DCRAW will take by default a white balance preset corresponding to lighting a grey car d with a standard D65 light source.
The correct linear values to be applied w ill vary from one camer a to another. To try to guess which values are to be used to get a correct w hite balance is not an easy task. However it is very simple to obtain the factors that correspond to each of the camera white balance presets which is a good start point. To do this we just need to take a shot with each of the presets and develop the r esulting RAW file w ith the -v -w commands that w ill display those numbers.
Find here a table of the RGB multipliers to achieve different white balance presets in the Canon 350D:
▪Default (D65 lamp): multipliers 2.395443 1.000000 1.253807
▪Tungsten: multipliers 1.392498 1.000000 2.375114
▪Daylight: multipliers 2.132483 1.000000 1.480864
▪Fluorescent: multipliers 1.783446 1.000000 1.997113
▪Shade: multipliers 2.531894 1.000000 1.223749
▪Flash: multipliers 2.429833 1.000000 1.284593
▪Cloudy: multipliers 2.336605 1.000000 1.334642
To specify that we do not want any white balance at all applied we will use the-r 1 1 1 1option. If doing so, we will be able to adjust later the white balance making use of the linearity of the histogram being this the best way to te st different white balance values until we reach one that satisfies us. However this is not a r ecommended way to apply the white balance, just t o calculate the multipliers. We should go back and use the calculated factor with the -r command as interpolaton Bayer
algor ithms are optimised to work on already balanced images.
HIGHLIGHT MODES
The -H option will decide DCRAW's behaviour concerning the highlights. We can distinguish two modes of oper ation according to the value range in w hich the w hite balance mul tipliers will be forced to be calculated:
-H 0 forces at least one multiplier be equal to 1, the rest will be greater or equal to 1
-H [1-9] forces at least one multiplier be equal to 1, the rest will be less or equal to 1
IMPOR TAN T: whatever the values that are set w ith the -r command, only the relative proportions between them are kept. It is the -H command who w ill decide if values are taken in one range or another.
With greater or equal to 1 multipliers we w ill guarantee that no tone artifacts will happen due to the white balance operation. As a drawback we will be in risk of blow ing partially (not all channels) or totally certain areas of the image due to saturation as we are increasing the image exposure with these values. -H 0 is the default mode if the highlight option is not set.
On the other hand less or equal to 1 multipliers will guar antee for all pixels that no channel that were not really blown in the RAW file w ill blow now because of the white balance scaling. The disadvantage is that depending on the -H [1-9] value chosen, we may have some colour artifacts in the areas that were originally blown in the RAW file.
All this sounds very complicated but it is not, let us take the previous example. In it w ith-w and -H 1 the follow ing multipliers were generated:
Scaling with black 256, multipliers 0.586287 0.421032 1.000000 0.421032
But if w e had used the option -H 0 w e would get these instead:
Scaling with black 256, multipliers 1.392498 1.000000 2.375114 1.000000
Both sets of factors keep the same channel proportions (0.586287/0.421032=1.392498/1.000000) corresponding thus to the same white balance; in particular this is the Tungsten white balance pr eset in the 350D. But there is a clear difference in w hat will happen when applying them: as -H 0 multipliers are greater or equal to 1 we could blow some channels that w ere not in the original RAW file. Anyone who has fiddled with the Temperature parameter in ACR will have noticed that moving this control may lead to blow n areas or make pr eviously blown areas not to be blow n any more. It's the same effect here.
But we must not think that for this -H 1 is always better. In fact is usually the contrary, we must use this last parameter value very carefully, and only if we are sure our image has no pixel at all blow n. In that case we can use -H 1 w ithout any undesirable effect. H owever everytime an image has some blown ar ea in the RAW file (and this includes metallic reflections, light bulbs, lamps,...) this is what we would get in case of using -H 0 and -H 1respectively:
Fig. 7 Result of development using -H 0 and -H 1 respectively.
In the first image we have probably blown some channels that were not in the original RAW file, but we have preserved the white colour of the blown highlights. On the other hand, the second image because of making use of multipliers less than 1 in the R and G channels, leaving B unalter ed, the proportions of those three channels in the blown ar eas have changed turning into an undesirable magenta cast. The phenomenon can be seen in the-H 0 vs -H 1histograms (look at the decomposition of blown areas in three peaks in different level locations):
Fig. 8 H istograms obtained when developing with -H 0 and -H 1 respectively.
The -H 2 value has a similar behaviour to -H 1 as will force multipliers to be less or equal to 1, but will fix this problems by applying a slight non linear correction in the highlights to guarantee a netrual (grey) colour in the blown areas.
To finish just to point that as the white balance means a scaling of all levels by a linear factor, we will be alt er ing their exposure, which could be adjusted w ith the linear exposure control we will discuss later. This is the r eason why the image developed with -H 0 (greater or equal than 1 multipliers) looked br ighter than the one developed with-H 1 (less or equal than 1 multipliers).
HIGHLIGHT RECOVERY
Values of -H between 3 and 9 activate different levels of highlight recovery making use of less or equal to 1 multipliers. These modes will take the tone from the non blown ar eas in the neighbourhood to fill the blown areas. The higher the par ameter value is set, the more effort will be done in emulating the surrounding tones and less guarantee to obtain neutral (grey) tones in the burnt pixels.
Follow ing is an example of the different performance concerning highlight recovery in an image w ith partially blown areas in the RAW file (bright areas in the girl's skin) compar ing ACR and DCRAW. While the first pushes the blown areas to neutral grey colours, in DCRAW we chose the -H 9 parameter to try to emulate the skin tones.
Fig. 9 H ighlight recovery compar ision ACR (left) vs DCRAW (right).
In this case DCRAW's strategy worked better as it imitates very well the ski n tones minimising the perception of undesired shiny areas. This does not mean that DCRAW will work better in any case at all. That will depend on each case accor ding to the characteristics of the image and its blown areas.
SENSOR SATURATION LEVEL
We commented in the introduction about the -k and -S commands for setting respectively the black and saturation levels of the sensor.
It is best not to specify the black level with -k since DCRAW will calculate it much better than us from the hidden sensor pixel s.
However the saturation level -S is closely related to the behaviour of the RAW developer in the highlights so it is very interesting to be able to set it since DCRAW, like any other RAW developer, will use a standard saturation value for each camera model, but this value could be unadequate for our particular unit or ISO set, or could simply be wrong in David Coffin's source code:
▪If our camera saturates the RGB channels in a level lower to that considered by DCRAW we could exper ience magenta cast issues with effects similar to those found in Fig. 7 because of channel missalignment.
▪On the other side if our camera saturates at levels higher than the standard value used we could be unnecessarily losing highlights information in the development process.
That is why it is a good idea to know exactly the saturation level of our camera's RGB channels and apply it to optimise the development. We shall take an example from a Canon 40D RAW file with blown highlight ar eas shot at ISO100:
C:\>dcraw -v -w -H 2 -4 -T portrait.cr2
Loading Canon EOS 40D image from portrait.cr2 ...
Scaling with darkness 1023, saturation 16224
Fig. 10 Magenta cast for using of a saturation level higher than real.
We can see DCRAW for this particular 14 bits camera applies a default saturation level of 16224 that produces wrong tones in the highlights. This is because this level value is too high for the tested camera. W ith the-D command we can analyse the RAW file to find out exactly at which level clips this sample of 40D:
C:\>dcraw -v -D -4 -T portrait.cr2
Fig. 11 Pure RAW histogram showing saturation level.
The camera under test saturates at level 13824, sensibly lower than DCRAW's default value. We develop again now making use of this new saturation level and we can see how the magenta tones disapperar without losing any infor mation at all:
C:\>dcraw -v -w -S 13824 -H 2 -4 -T portrait.cr2
Loading Canon EOS 40D image from portrait.cr2 ...
Scaling with darkness 1023, saturation 13824
Fig. 12 Level of saturation correctly applied.
The reason for the histogram not to reach the maximum is because we made use of a white balance with low er or equal to 1.0 multipliers which guarantees that no infor mation w ill get blown in the highlights producing a slight underexposure.
The point is that now DCRAW knew correctly which was the saturation level and forced a neutral tone (R=G=B) in the highlight areas thanks to the command -H 2.
Some cameras do not have exactly the same saturation point for all channels (this is pretty usual in N ikons), and some of them do not even saturate at a particular value but over a range of values (Fuji Super CCD R and Olympus files).
In these cases I have found that the correct w ay to proceed is to choose the lowest available saturation point, so that any higher value will be considered as saturated by DCRAW with the guarantee of neutral highlights.
For example let us take a RAW file fr om the Olympus E-510 w ith the follow ing RAW histogram in the blown highlights:
Fig. 13 Pure RAW histogram showing saturation levels.
DCRAW use s by default a saturation value of 3946 for this camera, producing slightly magenta highlights. When using a saturation point of 3648 in the development the blown areas become perfectly neutr al:
Fig. 14 Histograms after RAW development w ith different sat points.
Generally speaking we can conclude that when wrong tones appear in the highlight areas, typically magenta casts, is because of a software pr oblem in the RAW development. The RAW developer used is not applying a correct satur ation point and/o r an appropiate neutr al highlights strategy. It would not be strange that the problem disappears just by using a different RAW developer.
In the case of DCRAW the neutr al highlights strategy w ith -H 2 works perfect, but it is quite common to find default saturation points unadequate for some cameras. For them w e will have to calculate and apply it using the-S command.
To find out more about this important parameter please refer to T he RA W sa tu rati on l e vel tutor ial.
PHOTOSHOP ADJUSTMENTS OF THE DEVELOPED IMAGE
Once her e we should already have our RAW file developed and in 16-bit TIFF for mat, so it is time to get into Photoshop. When doing this PS w ill ask in which colour space the document must be open. If we told DCRAW to convert the data into some colour space PS w ill detect it and w e just need to tell PS to assign that colour space. If we did not, and the TIFF file has not colour managed (option -o 0), w e should tell PS to perform no colour management at all.
The ideal situation however is none of these options but to have the pr ofile of our camera, i.e. a profile generated after
calibr ating our own camera. In that case we would develop the RAW file w ithour any colour management (option -o 0), and in the file open operation we would assign our camera's own profile to it.
In the three cases the image data w ill be in linear for mat, but if w e have asked DCRAW to convert it to some colour profile the program keeps some information into the TIFF file which will inform PS that the data is in linear format so it will automatically display it w ith a corrected gamma. On the other hand if we did not convert to any colour space the image will display tremendously dark, do not panic. To correctl y visualize it we must go to Edit → Colour settings... and in the section 'Work colour spaces' choose RGB: 'C ustom RGB...'. A w indow w ill pop up and in the 'Gamma' value w e will introduce a 1. This will tell PS that the image has not been gamma corrected yet.
By doing this we are not altering the image data at all w hich remain linear exactly as DCRAW produced them. We just need to display the image's histogram to see how concentrated to the left it is.
If now we try to edit our image with curves for instance, it will be an impossible mission because of that concentr ation to the shadows. Unfortunately PS is not designed to do a proper linear edition, so its curves and the rest of tools do not have enough precision in the low end of a linear histogram to properly work on it. What we w ill be able to do now taking advantage of the linear state of the image data is:
EXPOSURE CORRECTION USING CURVES
This is a matter of having a clear concept of what digital exposure is: if our sensor is a linear device and as a result of this for every pixel it produces a level which is proportional to the amount of light captured, and our histogram is linear as well, t o apply on it an overexposure of +1EV we just need to multiply by 2 all image levels; and divide them by a factor of 2 to obtain an underexposure of -1EV. To perfor m this operation making use of a curve is as simple as:
Fig. 15 Curves for exposure correction.
To over expose by +2EV we would need a curve crossing the point (64,255), to underexpose by -2EV we w ould use the curve (255,64), and so forth. Of course all intermedium levels are also possible for an exposure fine tuning.
WHI TE BALANCE USING CURVES
To perform the white balance operation we will make use of the same linearity principle, it is just that in this case the curves will be applied independently over each of the 3 RGB channels to model the white balance multipliers we have been talking about. To find out the curve that models some particular multiplier we just need to do the conversion of it into the 0..255 range of the curve. For example the tungsten white balance that we studied had the follow ing RGB multipliers with-H 0: 1.392498。