DXOMARK offers impartial device quality benchmarks built on scientific assessments.
The French company’s state-of-the-art labs’ documented test protocols allow for the evaluation of the objective and perceptual performance of DSC and DSLR cameras and their lenses and battery life, as well as smartphone cameras, chipsets, ISPs and displays - all under real-world conditions.
DXOMARK also evaluates audio quality of smartphones and speakers. Results from hundreds of tests are aggregated as numerical values, providing sub-scores for important quality attributes and an overall quality index, which is the device’s DXOMARK score.
The consumer electronics community, manufacturers, and news media worldwide use the DXOMARK score as a benchmark for a device’s quality ranking.
Since the launch of its website in 2008, DXOMARK has quickly developed a reputation among manufacturers and consumers as a trustworthy source for those seeking product reviews based on scientific data.
The company began by testing DSLRs and camera equipment, later adding smartphone camera systems.
As touchscreen displays became the critical interface for smartphone operation, DXOMARK expanded its testing capabilities to cover display performance.
The DXOMARK display test lab opened in 2020, showing the company’s commitment to device benchmarking utilizing scientific testing, and employing up-to-date display test methods and equipment to assess quality according to the customer’s experience.
Display Quality as a Factor of User Experience
Display device and panel manufacturers generally carry out visual assessments and display calibration as an element of their quality control operations.
Usually, the display will be tested in dark labs, utilizing a measurement system aligned at a specific angle and distance relative to the display panel.
These parameters reduce the impact of variables such as view angle, ambient light and focus, isolating the display to characterize its color, brightness and other visual qualities under ideal circumstances.
This is not, however, how a user is likely to visualize their device or smartphone display.
DXOMARK takes a different approach to measurement. Nicolas Touchard, Vice President of Marketing at DXOMARK, explains, “We measure aspects like brightness, contrast, color, flicker - all sorts of parameters, which are physical attributes of the display. But we make assessments under real-life conditions. That is something quite different from the test conditions as defined in the industry standards as used by manufacturers.”
Customers view devices in a variety of ambient light conditions - in sunlight or at night, indoors or outdoors, and at a range of angles, distances and visibility conditions.
While manufacturer testing will measure the specifications of a display itself, DXOMARK offers objective insight into the quality of a display in real-world circumstances.
Image Credit: Radiant Vision Systems
All the standards right now are based on measurements made without ambient lighting, but that’s not a consumer use case. DXOMARK’s approach combines perceptual analysis and objective measurements, all being performed in conditions consistent with actual use cases. Our objective was to change the spirit of existing standards by doing measurement where nobody else wanted to go.
François Grandclerc, Display Quality Director, DXOMARK
Scientific measurements and repeatable test protocols are essential to DXOMARK’s testing procedures.
As a user’s experience of a device is essentially subjective, and subjective analysis alone cannot provide a benchmark, it is necessary to combine scientific measurements and perceptual analysis.
Each analysis should be carried out in circumstances that reproduce the user’s environment for the measurements that are most relevant.
Protocols, test setups and measurement systems ought to be rigorously defined and always carried out in the same way so that measurements are dependable, irrespective of who performs the testing and when the testing takes place.
If the instruments, the way the measurements are done, the setup, the protocols are not defined, you can’t have consistent data.
Nicolas Touchard, Vice President of Marketing, DXOMARK
The scientific data obtained contains photometric values measured from the display under a variety of circumstances. These values must be obtained in a controlled environment, restricting human intervention to ensure objectivity and to standardize the test process across devices.
DXOMARK’s testing methods utilize scientific metrology tools, software, visual inspection protocols and systems and robotics.
Measurement systems, including spectrometers and imaging colorimeters, are employed to obtain photometric data. For example, a display’s chromaticity coordinates (u´,v´ on the CIE 1976 chromaticity diagram), luminance value (Lv), contrast, brightness and color uniformity.
Robotics are utilized to automate test equipment and devices, including relative angle, alignment and positioning, corresponding with real-world test scenarios. A centralized software platform regulates all facets of the test procedure remotely, including:
- Operating measurement equipment and robotics
- Sending test patterns to the display via WiFi
- Adjusting variables such as ambient light
DXOMARK’s testing regulations set out the procedures and conditions applicable to measuring display characteristics - the performance parameters to be tested - according to each device use case, for example, how a device would be operated in the real world.
When we do our scoring, we test across many use cases like web-browsing, night reading, in-car navigation, gaming, watching videos. All of these use cases encompass what people will do with their smartphones. In each of these scenarios, we check attributes and do measurements of physical characteristics across these attributes, and we do this considering the ambient environment — the environment of the user.
Nicolas Touchard, Vice President of Marketing, DXOMARK
It is important to note that not every display attribute needs to be tested for every single potential use case. The test protocols employed by DXOMARK mean the most appropriate attributes are assessed for each setup.
In a use case such as web browsing, for example, display attributes like readability, artifacts, motion, color and touch ought to be analyzed as part of the user’s web browsing experience.
Fewer display attributes, like readability and artifacts, require testing in an in-car navigation use case, as the user is not usually interacting with the display.
A matrix of display quality attributes, testing conditions, and use cases that illustrate the complexity of DXOMARK display tests based on real-world conditions. Image Credit: DXOMARK
Components of a Scientific Display Test Lab
The demand for display quality benchmarking led DXOMARK to develop a new test lab committed to assessing the performance of displays. DXOMARK recruited a number of specialists to define a display quality benchmarking procedure.
A range of tools were selected to achieve the testing objectives, corresponding to the defined attributes and use cases. DXOMARK assessed a variety of display test instruments for the display test lab, including:
- Flicker measurement devices
- Imaging colorimeters
- Specialized lenses
- Spectroradiometers
- Gloss meters
- High-speed cameras
- Robotics to test touchscreen response
Assembling the pieces for automated display evaluation also involved the integration of robotics, measurement equipment and software into a controlled fixture. DXOMARK’s laboratory contains a custom-engineered test chamber called the Display Bench. It automates DXOMARK’s display testing under repeatable conditions.
The Display Bench features an LED panel to mimic ambient lighting conditions from darkness to daylight and robotic mounts for remote alignment of measurement equipment and smartphones.
Inside DXOMARK’s Display Bench, featuring a Radiant ProMetric® I Imaging Colorimeter, multi-axis robotic mounting equipment, and LED array to emulate lighting in use-case scenarios. Image Credit: DXOMARK
DXOMARK chose Radiant Vision Systems’ ProMetric® I-series Imaging Colorimeter and FPD Conoscope Lens for its automated visual inspection system.
As part of the Display Bench, the ProMetric® I-series Imaging Colorimeter measures the chromaticity, luminance and uniformity of the display in each test scenario. Conditions such as ambient lighting are varied for each display use case and attribute.
Every measurement utilizes numerous test images, such as colored images for chromaticity measurement, a white image for luminance and uniformity and a checkerboard contrast image for contrast and modulation transfer function (MTF).
Radiant Vision Systems’ equipment depends on imaging, allowing the ProMetric® system to capture the complete display in one instance and evaluate multiple attributes in just a few seconds.
The test images are pushed to the display in sync with software and camera operations, which are specific to each measurement scenario. This ensures fully automated testing without downtime.
The ProMetric I Imaging Colorimeter synchronizes test operations with images on the display for comprehensive evaluation in seconds. Image Credit: Radiant Vision Systems
Radiant’s FPD Conoscope Lens is utilized both inside and outside the Display Bench and paired with the same ProMetric® Imaging Colorimeter to capture a polar plot of the display as viewed to ±70° in one image.
This measurement characterizes chromaticity, luminance and changes in contrast as the display is viewed by a user from various angles.
In a typical use case, multiple users would be viewing movies or photos on the same device, and each user’s vantage point would differ depending on their position relative to the display.
A Radiant Vision Systems FPD Conoscope Lens measures the view angle performance of a display (left), providing a polar plot of luminance as seen across angles to ±70° (right). Image Credit: Radiant Vision Systems
DXOMARK’s display test laboratory also integrates equipment from Konica Minolta Sensing in Europe, which is Radiant Vision Systems’ sister company.
A Konica Minolta CS-2000 spectroradiometer within the Display Bench measures the color fidelity and brightness of the device over varying ambient light conditions and provides calibration data for the ProMetric® Imaging Colorimeter.
A Konica Minolta CM-25d spectrophotometer is also employed to evaluate gloss based on light reflected from the surface of the display.
Benefits of Radiant Vision Systems’ Equipment
The capacity of Radiant’s measurement and test equipment aligns with DXOMARK’s goals for user-focused display testing in a number of ways.
Radiant’s imaging colorimeters deliver objective, scientific data corresponding to colorimetric values. They are designed with tristimulus optical filter systems that measure light as it is perceived by the human eye.
During measurement, subsequent exposures of the display are captured through each filter in the camera.
The filters are weighted to human spectral sensitivity curves, with color matching functions standardized by CIE, guaranteeing the truest measure of color and light quality according to human perception.
Radiant’s ProMetric I Imaging Colorimeters measure color with an innate spectral response that matches standard CIE color-matching functions, quantifying display color as perceived by a user. Image Credit: Radiant Vision Systems
Radiant’s ProMetric® imaging systems are available in a variety of high-resolution sensor options to facilitate the increasing resolutions of modern smart device displays and even measure multiple displays at the same time.
Grandclerc explains, “For a smartphone now, you can have a 6-inch display with around 4K resolution. So, we needed a tool that shows us a luminance or color map with enough pixels to measure one device - at least. But in our approach, we measure four devices at the same time, and we need sufficient information measured across all devices. We needed the highest resolution available on the market, and it was Radiant.”
Radiant’s equipment offers DXOMARK the flexibility needed to meet a variety of test criteria.
For example, when testing view angle performance, the ProMetric® Imaging Colorimeter standard lens can be switched with the FPD Conoscope Lens to obtain angular measurements, offering, as Grandclerc comments, “the ability to switch from one to the other without buying another tool.”
Radiant ProMetric imaging systems can be paired with a range of electronically controlled and specialized lens options (like the FPD Conoscope) to accomplish a range of automated testing. Image Credit: Radiant Vision Systems
Radiant’s equipment also offers electronic control, as the aperture and focus settings of the ProMetric® imaging system’s lenses can be adjusted remotely using software.
Integrated with automated controls, particular lens settings can be applied for different device positions and alignments set by robotics to guarantee repeatable testing without the need for human involvement.
Grandclerc elaborates, “We can develop a smart algorithm to detect where the best focus is - same for the aperture. We wanted something that we could control electronically without putting an operator inside of the Bench to check everything - to avoid any mistakes, to be sure we are repeatable. Repeatability is key in our business.”
Each ProMetric® imaging system is controlled by software, capturing images and applying measurements in sync with automated processes. Radiant’s TrueTest™ automated visual inspection software is the control platform for each Radiant device.
It includes a programmable SDK and API to facilitate integration into fully automated testing processes and test benches.
Grandclerc comments on the importance of software integration, “It was maybe one of my first questions to Radiant - what can you do with the SDK to drive the camera?”
This capability allowed DXOMARK to incorporate the Radiant software and hardware functionality into its turnkey test system.
Equipment is only one piece of a comprehensive resolution, and global support from Konica Minolta Sensing in Europe and Radiant Vision Systems in the USA were both vital to DXOMARK’s test setup and operations.
Grandclerc says, “The quick and professional support from the team in the United States and here in Europe is something that is very important. The Radiant camera is a key tool for us, so when there is an issue, we need quick support.”
The development of the display test lab continued into 2020, in the midst of the COVID-19 pandemic. Members of the Konica Minolta and Radiant teams worked together to help DXOMARK meet its goals. “We even did some calibrations via video conferencing,” adds Grandclerc.
Beyond a Benchmarking Score
DXOMARK plans to extend its display quality testing in the future, incorporating further protocols and equipment to assess displays across new use cases. The company’s focus is not confined to employing scientific measurements to deliver benchmark scoring to consumers.
As an industry leader in measurement across the whole device component community, DXOMARK intends to provide consumers and manufacturers with a view of display performance that is relevant, easily understood and furthers device innovation.
DXOMARK’s test data facilitates like-for-like quality comparison across devices and delivers scientific metrics to both display panel makers and manufacturers that can aid the development of devices with improved performance. Such knowledge is vital for manufacturers seeking to provide customers with the highest quality user experience.
Touchard explains, “Our primary purpose is to make accurate, repeatable measurements and perceptual analyses that correlate with human perception, and which (at the end of the day) enable us to give a score - which is very consumer-oriented. Then our goal is to deliver very detailed reports of the characteristics of the device we’re testing serving our customers and providing very accurate and in-depth analysis so that they can understand how they are positioned relative to competition and how they can make improvements.”
Radiant’s ProMetric® Imaging Colorimeters and Konica Minolta’s systems, along with other leading display test and measurement equipment, allow DXOMARK to provide the scientific data needed to assist the consumer electronics industry in developing a greater understanding of device quality based on the user experience – the ultimate standard.
DXOMARK helps set the industry’s focus on a user-centric design by evaluating display performance under real-world conditions.
“DXOMARK develops and uses test methods to mimic smartphone users under different conditions, and the Radiant camera provides very accurate, repeatable, extensive measurements to test all photometric parameters that count for visual display quality.” - Nicolas Touchard, Vice President of Marketing, DXOMARK
This information has been sourced, reviewed and adapted from materials provided by Radiant Vision Systems.
For more information on this source, please visit Radiant Vision Systems.