AG DOK Camera Test 2016

→ deutsche Version

In the autumn of 2016 a team of five cinematographers and a colourist shot a comprehensive camera test in Berlin, Germany. Their intention was to examine a current line-up of cameras suitable for documentary work under realistic and matching conditions. Among the selection criteria for the cameras were log recording and interchangeable lenses. The shoot took place at exterior locations and with available light.

The selection included some low-budget cameras for research and/or inconspicuous shooting, as well as the latest mid range and high-end models: Blackmagic Pocket Cinema Camera (BMPCC), Sony α6300/α6500, Blackmagic Cinema Camera (BMCC), Sony α7S II, Blackmagic Production Camera 4K (BMPC4K), Blackmagic URSA Mini 4.6K, Sony FS7, Canon C300 Mark II, RED Raven, Panasonic Varicam LT, Arri Amira.

The test was designed to assess colour science, resolution, properties of the codecs and low light performance. The advantages of the ETTR exposure method are demonstrated, the image quality at high frame rates tested, and aliasing examined for internal 2K and 4K recording.

 

TEST SET UP

No artificial light was used. A location was selected where partial tree coverage created a strong natural soft source from one side and shade from the other, which was intensified by using a large flag for negative fill. Spot meter readings showed three stops contrast on the model’s face and a further 2 2/3 stops difference to the overcast sky.

 The aperture for the first take was determined by an incident reading with a Sekonic L558 Cine. The ETTR exposure was shot directly after the standard exposure to ensure the lighting conditions remained as similar as possible. The ETTR exposure was set by using on-board tools of the respective camera to bring the sky up to the level just short of clipping. The overcast sky ensured constant lighting conditions throughout the shoot. Adjustments to the exposure were made to compensate some slight variations.

 

The night exteriors were shot on a busy, well-lit street in the heart of Berlin. No movie lights were used, only the available, uncorrected light from shop windows and streetlights. The fine detail of the raindrops was especially challenging to the inter-frame codecs in the line-up, particularly in combination with underexposure. The low-light test consisted of 3 exposures: native ISO, 2 stops under and 4 stops under. The aperture remained constant at f2, underexposure was achieved by using ND filters and/or faster shutter speeds. Where available, the underexposure was compensated for by increasing the ISO. 

 

PROCEDURE

The main lens was a Canon CN-E 50mm T1.3 Cine. Voitgländer MFT Noktons were used  for the BMPCC and BMCC and a Sigma AF 18-35mm 1.8 DC for the slow-motion test.

Object distance, aperture and focal length remained constant throughout each set up except for the BMPCC and BMCC. Differences in sensor size (even between sensors that are nominally S-35) inevitably led to differing fields of view. ND filters and/or shutter speed were used to maintain a constant aperture across the varying native ISOs.

The test was graded in Assimilate Scratch using luminance and RGB curves, saturation controls and manufacturer LUTs only. Working on the assumption that it should be possible to produce a good image using primary controls only, colour, brightness and contrast were corrected to a point where the images could be compared with each other while preserving the individual characteristics of each sensor with its standard LUT applied.

UHD footage was scaled to HD with the Lanczos filter. Blackmagic RAW shots were debayered in Resolve with "sharpness = 0" and exported in their native resolution to 16bit DPX sequences.

Known errors:
- a cold battery interrupted the power supply to the BMPCC resulting in the loss of the night footage
- soft focus on the BMPC4K in the night footage
- URSA mistakenly shot HD instead of UHD in the night footage
- camera movement not ideal for the aliasing test
- soft focus on the α6300 rendered footage useless
- underexposure on the α7S II in the third step 2 stops under instead of 4

Camera settings:

*FS7 was operated in Cine-IE mode

 

ANALYSIS

25 frame UHD DPX sequences of each shot can be downloaded. 20Mbit/s HD Vimeo videos are provided to give a first rough impression.

The first Vimeo video is a blind test of the day/exterior set up. The cameras are identified in the description on the Vimeo page.

 
DPX-SEQUENCES
http://trollfilm.de/client-storage/agdok/AG_DOK_cameratest2016_Day.zip
http://trollfilm.de/client-storage/agdok/AG_DOK_cameratest2016_Night.zip
http://trollfilm.de/client-storage/agdok/AG_DOK_cameratest2016_Slowmotion.zip

  
DAY - BLIND TEST
BEST EXPOSURE NATIVE ISO

DAY
BEST EXPOSURE NATIVE ISO

 

DAY
MAKING THE GRADE - CAMERAS NOT MATCHED
BEST EXPOSURE NATIVE ISO

The following video demonstrates the composition of each grade. Cameras were not matched to each other to more clearly show the differences.

For all cameras without internal ND-filters Tiffen IRNDs were used, which are at least in parts responsible for some green casts.

 

ETTR vs STANDARD EXPOSURE

The split-screen shows 2 different exposures corrected to the same level: recommended native ISO as specified by the manufacturer on the left and ETTR (in most cases around 2 stops over native ISO) on the right.

The BMPC4K is excluded from this video because both methods resulted in the same exposure, ie. the sky almost clipped at native ISO.

Unlike celluloid which produces completely random noise, lossy digital formats can produce conspicuous unwanted compression artefacts.

The Expose To The Right method is used to shift the signal in logarithmic and RAW recordings away from the noise floor towards 100% white (on the right of the histogram, thus the name), to produce a signal that is as clean as possible without clipping relevant portions of the signal. Naturally ETTR does not work with steeper gamma curves that compress the highlights. Other workflow considerations such as consistency of exposure may influence the decision whether to use ETTR or expose in a traditional fashion. This test is designed to demonstrate the varying extent to which different cameras profit from the ETTR method.

Noise is the most difficult detail to compress. H.264-based codecs at low bit rates, as used in the Sony Alphas for example, definitely benefit from ETTR. Relevant parts of the image, light skin tones in particular, should not fall below 50%. Slight underexposure of images recorded with the XAVC-S codec can create blocky compression artefacts and colour blotches which have an extremely negative effect on the textures of faces. RAW recording and inefficient codecs like ProRes on the other hand reproduce noise much more naturally and are more forgiving of underexposure.

The Sony FS7 showed surprisingly little noise at standard exposure. The virtually noise free image quality of the Canon C300 MkII is presumably achieved using internal noise reduction which cannot be switched off and produces a rather soft image with less textured surfaces. The night shots illustrate this well.

All of the cameras showed a cleaner picture with ETTR. One could argue that for most cameras a non-ETTR recording constitutes underexposure.
 

 

SLPIT-SCREEN COMPARISONS

1) Amira, Varicam LT, C300II, FS7
2) BMCC, BMPCC, ɑ7S II, ɑ6300
3) Amira, FS7, ɑ7S II, ɑ6300
4) Amira, FS7, BMPCC, BMCC
5) Varicam LT, FS7, ɑ7S II, ɑ6300
6) Varicam LT, FS7, BMPCC, BMCC
7) C300II, FS7, ɑ7S II, ɑ6300
8) FS7, ɑ6300 , ɑ7S II, BMPCC
9) Amira, FS7, Ursa Mini 4.6k, Raven
 
 
1) Amira vs Varicam vs C300II vs FS7

2) BMCC, BMPCC, ɑ7S II, ɑ6300

3) Amira, FS7, ɑ7S II, ɑ6300

4) Amira, FS7, BMPCC, BMCC

5) Varicam LT, FS7, ɑ7S II, ɑ6300

6) Varicam LT, FS7, BMPCC, BMCC

7) C300II, FS7, ɑ7S II, ɑ6300

8) FS7, ɑ6300 , ɑ7S II, BMPCC

9) Amira, FS7, Ursa Mini 4.6k, Raven 

 

SLOW-MOTION

 

Cameras with a frame rate of at least 100 fps and min. 1080p:

 

The Varicam LT showed the most obvious drop in image quality. At 100fps there is a clear loss of resolution, even further at 200 with visible artefacts. Footage shot at 200fps can be improved by exporting from the Panasonic P2 viewer. (The Varicam LT footage was shot with a softer lens - Canon 16-35/4.0 instead of the Sigma 18-35/1.8 used on the other cameras)

At 100 fps the FS7 starts to show aliasing visible as colour artefacts.

Not surprisingly, the inter-frame codec of the α7S II cannot deal well with the moving water. As the frame rate increases, the image rapidly deteriorates.

The maximum HD frame rate for the C300 II, URSA Mini 4.6K and α7S II was lower than 200fps.

From 50fps upwards the URSA shows a strong magenta tint. Take 1 was shot through a .3 Tiffen IRND. The known green tint produced by the Tiffen filters likely compensated the magenta in the first shot. After correction there are still magenta blotches on the stones in the foreground in all 3 exposures, like those observed on the skin in the ETTR shot.

The Amira showed no apparent reduction in image quality and had the highest resolution at higher frame rates of all cameras in this test.
 

 

NIGHT

This test was primarily designed to assess the amount and quality of noise. The Vimeo videos only give a very rough impression as the additional recompression to 20Mbit/s H.264 of the uploaded footage does not allow for the noise to be adequately displayed. For closer examination, please refer to the DPX sequences, which can be downloaded using the links at the top to the page.

The three exposure steps are: native ISO (BMPC4K @ISO800); 2 stops under; 4 stops under:

At 4 stops under only the Varicam LT and the C300 MkII produce usable images. The same is probably also true for the α7s II, but the exposure was not altered correctly between the 2^nd and 3^rd shot.

With about twice the bit rate of Sony’s XAVC-I codec, the newer XF-AVC and AVC-Intra4K used by Canon and Pansaonic respectively showed finer noise than the Sony cameras. The relatively soft image of the C300 II is almost noise-free at 2 stops under and delivered the cleanest picture in the test field at 4 stops under, probably due to very effective internal noise reduction. The FS7 clearly showed artefacts at this exposure. The Amira and α6300 were usable up to 2 stops under.

Influence of bit rate and codec on detail and reproduction of noise: ProRes4444 @1106 Mbit/s (left), XAVC-I @250 Mbit/s (right)

Influence of bitrate and codec on detail and reproduction of noise: Prores4444 @1106 Mbit/s (left), XAVC-I @250 Mbit/s (right)

Influence of bitrate and codec on detail and reproduction of noise: cDNG RAW @ 720 Mbit/s (left) , XAVC-S @ 100 Mbit/s (right)

None of the Blackmagic cameras tolerated underexposure due to their fixed pattern noise (FPN). The BMPC4K and URSA showed FPN even at the first take (ISO 800, 1/25s). A current firmware update for the URSA supposed to improve the noise behaviour.

 

ALIASING 2K vs 4K

Over the last few years this shot has been repeated with a variety of cameras. In the video you will also see additional clips of older cameras (Canon C300, XC-15), the BMPCC with and without OLPF and with 2 different lenses, and the recently released Sony α6500. The RED was not available for the aliasing test.

The video should be played back 1: 1 at 1920x1080 to prevent additional aliasing produced by playback.

The test shows a tendency for modern 4K cameras to produce significantly more aliasing when recording 2K than at 4K. This trend was particularly pronounced with the α6300, α7s II, the C300 II and to a lesser extent the Varicam LT. It is advisable to operate these cameras in 4K or UHD.

In contrast to the C300 MkI or the Amira, scaling must be done in postproduction, which is a possible source of error. Incorrect scaling can also produce aliasing.

An alternative would be to use an external recorder that offers a higher quality downrez to 2K or HD.

The question arises why modern cameras do not at least use a simple Cubic filter for clean internal scaling. The small α6300 seems to be able to do this when it reads its entire 24M pixel sensor to produce a clean UHD image

 

SUMMARY

(Some of the following comments are anecdotal based on the experience of those involved in the test. Designed only to compare certain features, it was not within the scope of this test to provide a comprehensive overview of each camera, but we considered it useful nonetheless to share points made in discussions during the test in an attempt to indicate the relative strengths and weaknesses of the cameras tested.)  

 

BLACKMAGIC

The low price point of these cameras mean the buyer has to accept frequent production defects and partially unfinished new products that are later improved by firmware updates. Pixel errors, more or less pronounced fixed pattern noise or the new issue of magenta in the 4.6k URSA are only a selection of the problems.

In a preliminary test, the BMCC and BMPCC displayed significant sensitivity to infrared radiation. For this reason the cameras were equipped with IR cut filters for the main test.

In only 4½ years the camera products of this Australian postproduction device manufacturer, have gone from the aluminium block Cinema Camera 2.5K to the URSA Mini 4.6K with its ergonomic shoulder camera design and brought the aesthetic qualities of analogue film to low-budget digital shooting for the first time: Log gamma, high dynamic range, balanced colours, fine noise, lossless RAW and ProRes, features previously only found in Arri cameras.

 

BMPCC + OLPF

The Pocket’s images were the softest in the test. Without an optical low-pass filter their images are sharper but also show strong aliasing. Despite its poor codec the barely more expensive α6300 provides more image information with sufficient exposure.

Attractive aspects of the Pocket are its characteristic 16mm-like image quality, ProRes recording, the simple menu structure and very good focus peaking, superior to some high-end devices. The camera is a good match to the BMCC, but for those intending to use the FS7 (or better) in production the α6300 or the α6500 will be a better choice as the lack of resolution of the BMPCC caused by the small HD sensor becomes noticeable when compared on a large screen. 

 

BMCC

Despite its age, the Cinema Camera still has one of the best picture / price ratios in the test. Colour and dynamic range are still relatively good. The resolution is sufficient to keep up with the test field. In low light its poor sensitivity needs to be compensated by large aperture lenses such as the Voigtländer Nokton used in the test.

The SATA interface enables the use of SSDs which have the highest offload speed and best price/capacity ratio available and are physically more stable than conventional memory cards. RAW and ProRes recording ensure optimal workflow and noise performance.

Menu design is a big plus point on all the Blackmagic cameras.The larger touch display also simplifies menu operation and facilitates focusing thanks to excellent focus peaking.

 

BMPC4K

Both the day and night recordings of the BMPC4K fell significantly short compared to the test field, mostly due to its limited dynamic range and low native ISO of 400, which is probably a downside of its global shutter, a unique feature in this line-up. The higher pixel count reduces aliasing compared to the BMPCC and BMCC. The lack of an OLPF and excessive internal sharpening makes the image somewhat harsh. Colour separation was poor compared to the other cameras.

 

URSA MINI 4.6K

The URSA appears to offer the sharpest resolution of all cameras in the test, in part due to the lack of an OLPF, which results in coloured aliasing in 2K and 4K, not only in ProRes but also in RAW. Oversampling with a soft B-spline filter does not reduce this aliasing.

The ETTR exposure produced blotches of magenta around the brightest exposed skin tone which could not be corrected in post and are still visible even at standard exposure. Whether these artefacts have something to do with the documented magenta issue of the 4.6K sensor or are the result of a non-linear gamut could not be conclusively clarified. The high frame rate clips show a significant magenta problem as well.

 A lack of internal ND filters (now fixed in the recently introduced Mini Pro), poor low-light performance and impractical positioning of the air vents limits the usability of the URSA Mini for documentary work. As with all Blackmagic cameras, the menu and - a first for the Australian manufacturer -  the more ergonomic design as a shoulder camera are positive features.

 

SONY

α6300 / α6500

Resolution is the prominent feature of these small mirrorless still cameras. Despite having more than twice the number of pixels the sensor is read out completely and scaled down to UHD in real time. Like other 4K cameras it shows aliasing in HD recording. Limited by the low bit rate of the 100Mbit/s XAVC-S codec, internal recording shows compression artefacts even in the ETTR daylight shot.

The recently released α6500 adds stabilisation to the α6300 sensor. This can compensate some rolling shutter effects and also facilitates the use of lenses without their own stabilisation. The α7r II was excluded from the test as it offers the same features at more than double the price. The only advantages are a larger viewfinder, a brighter display and a headphone jack.

Its biggest drawbacks are the codec, which offers only moderate timeline performance even in Resolve, and the extremely complicated menu structure.

 

α7S II

The 4K sensor delivered a softer image than the α6300. It is currently the most potent full-frame video camera, but the 12MP sensor offers only moderate resolution for stills. It shows clear aliasing in HD recording. Due to its large pixel dimensions, the α7S II sees better in the dark than any other camera. Although its images are not free of artefacts at extreme ISO values, it can produce useable images up to about ISO 50,000. All the Sony Alphas are limited by the low (100Mbit/s) bit rate XAVC-S codec, compression artefacts were clearly visible both in the standard exposure daylight test and in the night-time shots. Like the α6500 it has a sensor stabiliser, which widens the lens choice. In contrast to the α6300/6500 it has a headphone jack. Once again, the complicated menu structure is a disadvantage of this Sony camera.

 

FS7

The FS7 currently dominates the entry-level production camera sector. Its colour science is not spectacular but nicely balanced, apart from the well-documented intensity of some red hues. Despite the 709TypeA(Arri) LUT these regularly still need to be separated from skin tones and desaturated in the grading. The sensor shows mediocre performance in low light and the H.264-based codec with its relatively low bit rate of max. 250Mbps in 4K produced visible compression artefacts. However with sufficient light the XAVC-I codec shows much fewer artefacts than XAVC-S used in the Alphas.

The FS7 has internal NDs, is lightweight and can be used as a shoulder camera. The nested menus with a multitude of functions that are insufficiently explained can be confusing and a potential source of error.

 

XAVC
The XAVC codecs are sensitive to noise and can show compression artefacts even in the shadows of normally exposed shots with sufficient light. These are particularly prominent with the XAVC-S codec used by Sony’s Alpha line. ETTR is the best way to reduce this, especially when using SLog3.

In addition to ETTR, recording to an external device to Prores is recommended, to reduce compression artifacts even further.

 

CANON C300 MkII

The C300 MkII disappointed somewhat due to its relatively soft picture and an unusual colour palette, as seen through the LUT supplied by the manufacturer. In a pre-test all gamuts and relevant matrices offered by the camera were examined, but none of the settings, even the one that should simulate the colours of the popular predecessor C300 MkI, were convincing. A new manufacturer LUT and some simple adjustments to the RGB curve would bring the characteristics of this camera into more familiar territory.

Compared to the FS7 the Canon offers a better codec and overall better performance, especially in low light. Autofocus was not tested, but a team member described it as excellent.

The biggest downside of the camera is its strong aliasing at 2K.

The Mark II retains the familiar design of its predecessor and cannot be considered a shoulder camera - an exclusion criterion for some DPs, whereas for others the compact form may be considered particularly advantageous.

 

RED RAVEN (SKIN TONE OLPF)

The Raven appears to be an attempt to lower the entry threshold to the modular world of RED cameras. Its future, however, is uncertain as RED seems to have given up on the product. Jarred Land, president of RED Digital Cinema, stated in 12/2016, “I made a big mistake with Raven... it is just too good for the price and we can't make them fast enough and I have a feeling we never will.”

At € 22,000 for a basic tripod set up (ie. without viewfinder or shoulder pad) the price can hardly be justified with regard to its performance. The small 4,5K widescreen Dragon sensor produced good results in daylight with regard to colour reproduction and resolution, but at night it barely outperformed the FS7.

RED’s wavelet codec is a lossy RAW- format with variable compression rates. While its primary advantage is its ability to decode the material at lower resolutions in real-time without the necessity to transcode it, this is also its weakness for filmmakers who want to edit at full resolution. Despite the recent GPU support of some applications, such as Scratch, Resolve or Premiere the codec still requires the proprietary decoder card from RED for full debayer quality.

As with the URSA, the ventilation slots at the top of the camera seemed problematic during our rainy night shoot. The lack of internal NDs is also an exclusion criterion for many documentary shooters.

 

PANASONIC VARICAM LT

The LT is clearly the price/performance winner in the high-end segment. Purpose built for hand-held shooting, it has a good viewfinder, is reasonably well balanced and lightweight.

The Panasonic LUT produces attractive, balanced colours with a touch too much luminance in the red channel. The dual native ISO of 800 and 5000 performed very well in the night shoot, the image quality surpassed that of the Amira at 2 stops underexposure.

With a bit rate of only 388Mbit/s the AVC-intra4K codec is almost twice as efficient as ProRes 422HQ. In the subjective opinion of the test crew, noise appeared only slightly more digital-looking than that of the Amira.

Image quality drops significantly at 100fps and further at 200fps.

Featuring excellent image quality, especially in low light conditions and good design at a reasonable price, this camera will be a serious alternative to the Amira for many documentary filmmakers.

 

ARRI AMIRA

Featuring the same sensor as the Alexa, which has been the benchmark for digital imaging since it was introduced in mid 2010, the Amira is optimised for single person hand-held shooting. The unmatched dynamic range, smoothest highlight roll-off and degree of colour separation with no visible aliasing for which this camera is well known were confirmed in this test. Fine noise in the shadows gives the picture an analogue texture, and it was generally agreed that the Amira offers the most natural look. The Varicam comes quite close, however, and surpasses the Arri beyond ISO 1600. Whereas all the other cameras sacrifice image quality with increasing frame rates, the resolution of the Amira remains constant up to its maximum of 200 fps.

UHD resolution was critically monitored by the test crew, as many in the industry are uncertain whether the Amira’s 3.2K sensor readout can produce a competitive UHD image. Unfortunately the sightly soft focus and wider field of view of the Amira did not allow for a perfect comparison to the Varicam with its 4k sensor. Factoring in also the higher amount of aliasing of the Panasonic, the differences in resolution appear not to be significant in this test.

The relatively inefficient DNxHD and ProRes codecs require more storage space, but also contribute to the analogue quality of the image and reproduce fine detail without visible compression artefacts. Compatibility with all editing and finishing systems and an easy workflow even on weaker systems is guaranteed.

 

TEAM

Left to right: Frider Unselt, Rasmus Sievers, Felix Trolldenier, Carolin Rähmer, Tobias Mennle, Henning Stirner, Christopher Rowe, Andrea Gatzke

 

Organisation, Postproduktion, Evaluation: Felix Trolldenier AG DOK
Organisation: Helge Renner AG DOK
DoP: Christopher Rowe BVK
Camera operators: Tobias Mennle , Rasmus Sievers, Henning Stirner, Andrea Gatzke
Camera assistent: Frider Unselt
Modell: Carolin Rähmer

Text: Felix Trolldenier, with support of Helge Renner and Christopher Rowe

Camera operators pre test: Eberhard Spreng AG DOK, Richard Heidinger AG DOK, Robert Löved (SlashCam), Sabine Streckhardt BVFK
Model pre test: Magdalena Mrigida

Andrea Gatzke studied photography in Hannover, camera assistant in Berlin and cinematography at the Filmakademie Baden-Württemberg. Since graduating in 2005 she works as a free lance camera woman. www.andreagatzke.de

Tobias Mennle has been making films since he was 14. A qualified biologist, he works mostly on international natural history films as cinematographer, director and producer. His specialty is observing naturally occurring animal behaviour in inaccessible places worldwide and at home. www.tmfilmpro.com

Helge Renner AG DOK is a filmmaker specialising in documentaries and corporate films. A trained photographer, his primary function is cinematographer, but his skill set also includes directing, producing, sound and editing. www.vimeo.com/helgerenner

Christopher Rowe BVK relocated from his native England to Berlin in 1983. His training includes a post graduate course with Michael Ballhaus ASC in Hamburg. He shoots both fiction and non-fiction worldwide, mostly for German television. www.christopher-rowe.de

Henning Stirner received his camera assistant training in Berlin and studied cinematography at the Filmakademie Baden-Württemberg. Since graduating in 1997 he shoots fiction and non-fiction, commercials and music videos. www.henningstirner.de

Rasmus Sievers worked his way up through the camera department and also filmed for German TV stations before studying cinematography in Potsdam-Babelsberg. Since graduating in 2008 he shoots mostly documentaries and commercials. www.rasmus-sievers.de

Felix Trolldenier AG DOK is colorist and online editor with a background in physics and film studies. After graduation in 2010 he worked in NYC in post-production of feature documentaries. In 2012 he built his own studio in Berlin for color grading, finishing and on-set services for independent cinema and art projects. www.trollfilm.de

 

Thanks to SlashCAM for lenses for the pre test, to Anna Kersting and to our sponsors, who provided equipment:

 

COPYRIGHT

Provided material (images und videos) may only be used privately and published only with permission of the AG DOK.