Welcome to our comprehensive guide to Quality of Experience in video technology.
If you’re looking for the following:

• Why QoE in Video Matters Today
• QoE and Revenue Loss Statistics
• Video Encoders and QoE
• Quality Assessment Scoring Methods
• Encoding Choices and QoE

Then  you are in the right place.
Let’s get started!

Why Quality of Experience in Video Matters Today

Content owners are investing large sums of money on their premium content, so it is more important than ever they invest in maximizing the quality upon delivery as well.
Content needs to be prepared and streamed in impeccable quality in order to satisfy viewer expectations, and those expectations are getting increasingly demanding. Recent events have shown us that viewers are not shy to air their grievances when things don’t look or stream right.
From major sporting events to big-budget series, more of us expect to experience the fully immersive deep colors and vivid images promised by HDR and the 4k experience. When it isn’t forthcoming, users are quick to voice their displeasure.
Game of Thrones fans may recall the final season’s quality fiascos. Social media lit up with posts regarding the HBO streaming experience. Rather than mentions of epic battle scenes or plot twists, viewers took to social media to complain of difficult-to-see scenes, washed-out colors, and image artifacts — especially apparent on large television screens.

The key take-away: Great content is no longer enough to keep audiences engaged!

QoE and Revenue Loss Statistics

Industry studies show that as many as 33% of users leave a stream due to poor streaming quality. Verizon estimates that OTT video services delivering average or poor-quality experiences account for as much as 25% loss in revenue.
With the explosion of VOD platforms competing for our attention, it is vital that quality is not be a reason for your audience to churn or tune out. 

So, what’s changed?

Only a decade ago, viewing high definition video online was a luxury, now high video quality has become a commodity and continues evolving.
High Definition and 4K resolutions are currently the industry standards (with 8K becoming more widespread).
In addition to high pixel count, the quality of pixels has consistently improved over the years as well. Today, pixels are comprised of more bits – which translates to more vibrant colors and details, enabling HDR technologies. 
Audiences agree that these technologies offer a superior viewing experience over past media formats. However, implementing cutting-edge media technology is not always seamless. 
Online streaming has proved to be one of the most challenging applications for new media formats.
Some of the challenges that content providers face include:  

Lack of standardization

Device targeting with a given codec is necessary to optimize quality, and it becomes increasingly difficult with fragmented software & hardware codec support. Encoders must prepare content in a number of HDR formats (HLG for broadcast, HDR10 for streaming, Dolby Vision for streaming, etc) to reach viewers across devices.

Authentic 4K/8K experiences start with video production 

Unless the entire upstream pipeline consists of a native 4K environment and downstream devices support 4K, image scaling is required.

Last-mile bandwidth limitations

Even with faster home and mobile networks and more efficient codecs, compressing 4K, 8K and HDR video into a size which can be efficiently streamed at the last mile is difficult without compromising quality. 
Delivering a high-definition stream free of network interruptions (rebuffering) is no longer enough to satisfy viewers and subscribers. Today’s viewers expect immersive video experiences.
They want to take advantage of the rich features their screens and devices support — 4K, 8K, HDR, and next-generation audio. 

Video Encoders Responsibility

Modern-day Adaptive Bitrate (ABR) video players are resilient to bandwidth fluctuations, but video encoders must be able to produce the clearest possible images at the most efficient bitrates to satisfy Quality of Experience (QoE) expectations.
Whether a talking head in a newsroom, an action-packed war scene, or a close-up of the game-winning goal, encoders must be flexible enough to maintain pristine quality in all content scenarios.
But exactly how much does quality matter and how should content owners identify and address quality problems?
Many still believe the key to high-quality video is higher bitrates.
That approach does not leverage the power of codecs or consider limited bandwidth audiences are faced with (nor is it cost-effective).
Either by increased delivery costs or quality-related churn; wrongly equating bitrate for quality will have a negative impact on the bottom line.
The most effective approach to optimizing encoding profiles will ensure visual perceptual quality is not compromised (improved QoE), bitrates are reduced for better delivery and availability performance (improved Quality of Service), and storage and delivery costs go down (improved economics).

(State-of-the-Art) Quality Assessment Scoring Methods

A viewer’s rating of quality is important for experience metrics, however, it’s not a reliable measurement of quality for production and distribution stakeholders such as service providers or network operators.
A standard consumer can provide valuable insight into subjective quality assessment, but subjective measurements often lack scientific objectivity and scalability.
For this reason we focus on repeatable and objective quality measurement methods.
Although there are many measurements, we have three identified primary methods “plus” an extension that are ideal for objective quality assessments:

Video Multi-Method Assessment Fusion (VMAF):

One of the latest metrics adapted by the streaming community is Netflix’s Video Multi-method Assessment Fusion (VMAF). It predicts subjective video quality based on reference and distorted video sequences.
The metric can be used to evaluate the quality of different video codecs, encoders, encoding settings, or transmission variants.

Structural Similarity (SSIM):

The structural similarity index is a method for predicting the perceived quality of digital television and cinematic pictures, as well as other kinds of digital images and videos.
As its name indicates: SSIM is used for measuring the similarity between two images.
The SSIM index is a full reference metric; in other words, the measurement or prediction of image quality is based on an initial uncompressed or distortion-free image as a reference.
SSIM is designed to improve on traditional methods such as peak signal-to-noise ratio (PSNR) and mean squared error (MSE).

SSIMPLUS:

The SSIMPLUS Viewer Score is a family of algorithms invented by the team that built SSIM, based on two decades of research & development after inventing SSIM, and is a commercially available tool.
SSIMPLUS measures video experience from content creation to consumption in an Apples-to-Apple fashion and assigns scores between 0–100, linearly matched to human subjective ratings.
SSIMPLUS also adapts assigned scores by intended viewing devices, viewer types, such as Studio, Expert, and Typical viewers; thereby comparing video across different resolutions, frame rates, dynamic ranges and contents.
According to its authors, SSIMPLUS achieves higher accuracy due to its ability to assess all commonly occurring impairments and higher speed than other image and video quality metrics. The Viewer Score is validated using publicly available and commonly used subject-rated datases. A visual comparison with other objective quality assessment approaches is available here.

Peak Signal-to-Noise Ratio (PSNR):

The Peak Signal-to-Noise Ratio is a video quality metric that combines human vision modeling with machine learning.
PSNR is most commonly used to measure the quality of reconstruction of lossy compression codecs (e.g., for image compression).
When comparing codecs, PSNR is an approximation of the human perception of reconstruction quality.
Generally speaking, a higher PSNR indicates that the reconstruction is higher quality, in some cases it may not.
However, in cases where the codecs and/or content are different, the validity of this metric can vary greatly; therefore, you should be extremely careful when comparing results.
PSNR is a well-established method especially in research and also within the industry when measuring at scale mainly given its simplicity.
But it is also known to not correlate well to human perception. Other metrics, such as SSIM or VMAF better correlate with subjective scores but can be quite expensive and time-consuming to compute.
In the end, it is up to an individual encoding professional to decide which method to use testing video quality.
There are additional quality metric scores such as MOS and DMOS, and the industry is still in the process of determining the gold-standard of objective quality measurement.

Encoding Choices

With objective quality assessment tools in hand, it is now easier for you and encoding professionals to evaluate and select which encoding technology will best suit your content delivery (and quality) needs.
So which technologies currently exist in the market and what are the ways that to encode (especially for quality)?

Codec Support Selection – it’s a multi-codec world! 

Video Codecs:

H.264/AVC

The industry standard for video compression – designed and maintained by the Moving Pictures Expert Group (MPEG) since 2003. According to our 2019 Video Developer Report – is currently used by over 90% of video developers 

H.265/HEVC

MPEG’s most recent video compression standard – designed and maintained since 2013 – offers 50% higher efficiency than its predecessor

VP9

Google’s royalty-free video compression standard – designed and maintained since 2013. Mostly used on YouTube, but otherwise does not offer full device reach. Great for saving on CDN/Bandwidth costs!
AV1
Another open-source code, designed by the Alliance for Open Media (AOMedia), a conglomerate of video tech giants like Google, Facebook, Netflix, Amazon, Windows, and Bitmovin.
AV1 offers 70% better compression rates than H.264 but is currently limited by application within browsers and devices. However, this is slated to change within the coming two years.

Next generation codecs:

VVC

Versatile Video Coding (VVC), is also called H.266, ISO/IEC 23090-3 and MPEG-I Part 3 is a video compression standard finalized on 6 July 2020 by the Joint Video Experts Team (JVET) of the VCEG working group of ITU-T Study Group 16 and the MPEG working group of ISO/IEC JTC 1/SC 29.

EVC

MPEG-5 Essential Video Coding (EVC) is a current video compression standard that was completed in April 2020 by decision of MPEG Working Group 11 at its 130th meeting. The standard is to consist of a royalty-free subset and individually switchable enhancements.

LCEVC 

Low Complexity Enhancement Video Coding (LCEVC) is a ISO/IEC video coding standard developed by the Moving Picture Experts Group (MPEG) under the project name MPEG-5 Part 2 LCEVC

Benefits of single codec usage vs Multi-codec support

Even though H.264 is ubiquitous and widely supported at the hardware level, it is much less efficient than next-generation codecs in terms of compression rate.
By encoding your videos using a multi-codec approach you can aim to double the quality while still reducing your bandwidth consumption without compromising device reach.

This graphic shows you how to perform an encode

Now that you’ve selected which codecs you’ll be using, the next step is to determine how you’ll encode your video content. To retain the best quality during conversion, we’ve determined that multi-pass encoding is the best option; below you’ll find the types of multi-pass encodes:

2-Pass Encoding

A file is analyzed thoroughly in the first pass and an intermediate file is created. In the second pass, the encoder looks up the intermediate file and appropriately allocates bits, therefore, the actual encoding takes place during the second pass.

3-Pass Encoding

Similar to 2-Pass encoding, 3-Pass encoding analyzes the video three times from the beginning to end before the encoding process begins. While scanning the file, the encoder writes information about the original video to its own log file and uses that log to determine the best possible way to fit the video within the bitrate limits the user has set for the encoding process.

Per-Title Encoding

A form of encoding optimization that customizes the bitrate ladder of each video based on the complexity of the video file. The ultimate goal is to optimize towards a bitrate that provides just enough room for the codec to encapsulate information to present a perfect viewing experience.
Another way to consider it, is that the optimized adaptive package is reduced down to contain the exact information for optimal viewing quality. Anything beyond the human eye’s ability to perceive is stripped out. 
Some of the true magic behind encoders is their ability to choose how to implement or tune a given codec.
In some cases, encoders allow users to configure and optimize codec compression settings, like motion estimation or GOP size and structure. 
It goes without saying, but the best method to ensure top quality, even through an encode, is by supplying high-quality sources; starting with pristine quality videos and using best practices for signal acquisition/contribution. 
More Readings:

• What is DRM and How does it work?
• Video Technology Trends
• Multi-Cloud Video Encoding
• Encoding Definition and Adaptive Bitrates 
• HTML5 Video Tag Guide 
• VOD Platforms 
• HEVC vs VP9: The Battle of the Video Codecs

Follow Bitmovin on Twitter: @bitmovin

Did you know?

Bitmovin has a range of video streaming services that can help you deliver content to your customers effectively.
Its variety of features allows you to create content tailored to your specific audience, without the stress of setting everything up yourself. Built-in analytics also help you make technical decisions to deliver the optimal user experience.
Why not try Bitmovin for Free and see what it can do for you.
We hope you found this guide useful! If you did, please don’t be afraid to share it on your social networks!

The post Quality of Experience (QoE) in Video Technology [2022 Guide] appeared first on Bitmovin.

SDR vs HDR – to Convert or not to Convert, that is the question

So what do you do with all of your existing SDR content? Can it be converted to HDR? In a few ways, yes, though results will vary depending on the chosen approach. There’s also the old adage “quality in, quality out” or the less optimistic “garbage in, garbage out” meaning that even with the best solutions, the quality of the source material is going to impact the quality of the final product. Existing compression artifacts or camera noise have the potential to be magnified and become more noticeable when upconverted, requiring extra “clean-up” pre- or post-processing steps. Thus one must ask the question, should SDR content be converted to HDR in the scenarios that it could? Even if an SDR source file has been successfully converted to the HDR specifications, it might still look exactly the same, depending on the technique applied, so let’s take a closer look at what needs to be done and the available options to create noticeable and worthwhile upconversions. 

How does SDR become HDR?  

There are two things that need to happen to transform SDR video into the HDR domain. First, the color gamut has to be widened to the desired color space, usually from the standard Rec. 709 to the wider Rec. 2020, which represents a spectrum of color possibilities much closer to the capabilities of what the human eye can detect. 

Direct Mapping vs Up-Mapping 

The simplest approach to converting SDR to HDR involves directly mapping the equivalent color and brightness values, essentially encapsulating the SDR signal in an HDR container, along with associated metadata. This is the most computationally and cost-effective method to complete the technical conversion to HDR, but the video being identified as HDR will look the same as the SDR source which can lead to an inconsistent and disappointing end-user experience. 
The preferred method from a quality perspective is known as up-mapping or inverse-tone mapping and creates an “HDR-look”, allowing SDR clips or shows to blend into an HDR production or platform. This is achieved through the use of tone-mapping filters or Lookup tables (LUTs) that have been extensively tested and calibrated to accurately recreate and enhance SDR content in the HDR domain. Up-mapping is more computationally complex, and thus more expensive than direct mapping, but the visual difference is significant and necessary for a true SDR to HDR workflow. 

Improving Quality AND Experience

Streaming services with a mixed catalog of old and new, SDR and HDR, should strongly consider up-mapping their SDR content to match the look and format(s) of their HDR content. It’s the best way to ensure the highest quality, most consistent and enjoyable viewing experiences across their entire library. In a crowded market, it will elevate and differentiate their service. 

The Future is in HDR 

We are living through a transitional period of mixed content and no doubt will eventually reach a point where everything is being produced in HDR, there’s universal playback support, and all of the legacy SDR videos deemed worthy will have been upconverted. Streaming industry analyst Dan Rayburn sees HDR as a key advantage and predicts we’ll see the fastest rates of adoption in streaming services over the next 2 years. It’s really not a matter of if, but when, so any content that has long-term appeal should be upconverted to HDR as soon as possible to maximize the lifetime value of that process. Keep an eye out for our upcoming post about how to streamline your HDR encoding workflows with Bitmovin, including examples for upconverting SDR to HDR with the Bitmovin encoding API. 
In the meantime, if you want to find out more about SDR vs HDR and the best upconversion approach for your service, get in touch today or check out one of these other great resources:
[Blog Post] HDR vs SDR: Why HDR Should Be Part of Your Video Workflow
[On-Demand Webinar] Deploying HDR for Streaming Services: Challenges and Opportunities
[On-Demand Webinar] How to Enable and Optimize HDR Encoding

The post Raising the Bar on Quality: How to Upconvert SDR to HDR appeared first on Bitmovin.

Quality Matters – Factors that affect viewer experience

There are countless factors that can negatively affect your subscribers’ experience – luckily most of them can be resolved with a simple combination of accurate meta-data and specific dashboard inputs. For your convenience we’ve organized the top six factors that will make the most positive impact on your workflow:

Black Bar removal

Perhaps one of the most noticeable factors that may affect a viewer’s Quality of Experience is the addition of those pesky Black Bars that appear on both sides of a video player during playback. Bars or letter-box artifacts occur when an asset of a non-conforming aspect ratio is introduced somewhere in the workflow. A mezzanine asset typically would not see this, but where content has moved through upstream systems – the likelihood increases. Typically Bitmovin’s tools initiate a black bar removal if either an asset’s technical metadata requires it or the Ooyala Flex ingest path determines it necessary.  
For correction, Bitmovin encode contains a cropping filter, which can be controlled through the Ooyala Flex Media Platform to remove the required pixels or frame percentage, thus correcting the image. You can see this process below:
[bg_collapse view=”button-blue” color=”#f7f5f5″ icon=”eye” expand_text=”Show Bitmovin API Reference” collapse_text=”Hide Bitmovin API Reference” ]
CropFilter{

[/bg_collapse]
Some of the aspect ratio issues that may come up within the Pixel Aspect Ratio (PAR), Storage Aspect Ratio (SAR), and/or Display Aspect Ratio (DAR). The ideal AR should be as follows:

• PAR: the aspect ratio of the video pixels themselves. For 576i, it’s 59:54 or 1.093.
• SAR: the dimensions of the video frame, expressed as a ratio. For a 576i video, this is 5:4 (720×576).
• DAR: the aspect ratio the video should be played back at. For SD video, this is 16:9.

Technical metadata detection within Ooyala Flex will, in most cases, correctly determine the aspect ratio characteristics, allowing the encoding profile in Bitmovin’s dashboard to be adjusted automatically. These are all characteristics of a video asset. Getting it wrong either in detection, encoding, or playback results in squashed video playback. 

Colour-Correction

Bitmovin’s encoder contains a powerful set of color space, color range, and color primary manipulation logic. While not a full color-grading solution, the encoding workflow can easily be modified to correct for color issues commonly found in mezzanine formats.
[bg_collapse view=”button-blue” color=”#f7f5f5″ icon=”eye” expand_text=”Show Bitmovin API Reference” collapse_text=”Hide Bitmovin API Reference” ]

[/bg_collapse]

Deinterlacing

Interlacing or deinterlacing can be used to drastically improve the visual performance of your fast-moving content. When manipulating these aspects of the encode it’s important to have a full view of the input asset making the decision to trigger these ‘filters’ for either content of a particular source, a particular technical metadata characteristic, or as part of a human-driven QA process.
[bg_collapse view=”button-blue” color=”#f7f5f5″ icon=”eye” expand_text=”Show Bitmovin API Reference” collapse_text=”Hide Bitmovin API Reference” ]

DeinterlaceFilter{

[/bg_collapse]

Conformance (FPS)

Source content is unlikely to always be captured at the same frame-rate. US-sourced content can range from 29.97 FPS to capture rate of 25/50 FPS – or even faster! Although the normal role of the encoder is to conform all inputs to a given frame-per-second, especially when ad-insertion is used, there are also some use cases where certain content coming through certain workflows need to maintain different (and higher) values.
[bg_collapse view=”button-blue” color=”#f7f5f5″ icon=”eye” expand_text=”Show Bitmovin API Reference” collapse_text=”Hide Bitmovin API Reference” ]

ConformFilter{

[/bg_collapse]

Post/Precuts

Rarely will content start exactly where you want it to, be it color-bars or lead-in titles, or simply a longer form recording that runs too long. Being able to clip out a beginning by X seconds and clip out an end at Y seconds can avoid the costly exercise of offlining content for craft editing. The Ooyala Flex-powered workflow can control clip-in and clip-out parameters, automating ingest where required.

Audio-Levelling

Some content is loud, some content is quiet. The audio-filter allows all or selected content to have its volume adjusted, thereby creating a uniform viewer experience. 
[bg_collapse view=”button-blue” color=”#f7f5f5″ icon=”eye” expand_text=”Show Bitmovin API Reference” collapse_text=”Hide Bitmovin API Reference” ]

AudioVolumeFilter{

[/bg_collapse]

How will you overcome Quality of Experience issues?

To summarize there are six key factors that often affect a user’s QoE: black bars, poor coloring, incorrect lacing for visual quality, non-optimized FPS, content length, and inappropriate audio volume. So, how do you overcome the most glaring Quality of Experience issue: aspect ratio-related issues?

How to detect Aspect Ratio-related Quality of Experience issues

When it comes to content preparation, everything really starts with the Ooyala Flex Media Platform, extracting Technical Metadata from incoming media. This critical step extracts information such as format, framerate, frame size, colour space, D.A.R., P.A.R., codecs, bitrates, and specific details for codecs in use (e.g. GOP structures, profiles, audio sample rates and bit-depths), audio track counts, timecode start time and duration. 

The evaluation of the technical side of our media is a great start, but what about validation of the media essence?

The Ooyala Flex Media Platform incorporates a number of tools that allow clients to effortlessly integrate with external products; a perfect example of this integration is for automated quality control, such as Dalet AmberFin. Automated QC reports from external products can be analysed by Ooyala Flex, workflow orchestration can take remedial action to correct any QC issues, and submit corrected media to automated QC again to ensure compliance.

• Contact Dalet
• Contact Bitmovin

The post Quality of Experience Issues: Levelling the Content Creation Playing Field (ft. Dalet) appeared first on Bitmovin.