That changed when Apple launched the Apple Vision Pro, announcing that unlike Meta Quest and other headsets, their new device would take advantage of MV-HEVC for immersive video experiences. In this blog post, we’ll explore what MV-HEVC is, its potential for enhancing streaming experiences and how to get started. 

What is MV-HEVC?

MV-HEVC stands for Multiview High Efficiency Video Coding, an extension of HEVC that was added to the second edition of the standard in 2014. It’s designed to support the efficient encoding of multiview video content captured from multiple viewpoints, often to create stereoscopic (3D) effects or spatial video experiences for virtual reality (VR) and augmented reality (AR). 

Doubling the encoding and bandwidth requirements for multiple viewpoints could potentially create buffering and playback issues, but MV-HEVC enables the efficient compression and storage of stereoscopic content, reducing the bandwidth required for streaming or the file size needed for storage without compromising the video’s quality.

In short, MV-HEVC allows the encoding of multiple views of the same scene in a way that preserves video quality while keeping the bitrates manageable. This makes it a good fit for 3D, AR and VR applications that require a lot of real-time data processing. 

How MV-HEVC works

Before getting into how MV-HEVC works, let’s take a quick step back to the basics of video encoding. Temporal compression is a technique for reducing file size that is common to all major video codecs. Unless there is a scene change, individual frames of video are usually not that different from one frame to the next. Temporal compression exploits that fact and reuses data where it can, saving some bits from being encoded and shrinking the file size. 

This is done by encoding different types of frames that require less data to reconstruct for playback. I-frames are fully encoded frames that serve as anchor points, while P-frames (Predictive frames) can reuse data from frames that came before them. B-frames (Bi-direcional predictive frames) can reuse data from frames both before and after them. If you’re interested in learning more about some of the fundamentals of video encoding, check out this guide. 

I touched on all of that because a key benefit of MV-HEVC is that it is also able to take advantage of the commonalities across multiple camera angles or views. In the cases of immersive and 3D videos that are created with different views for the right and left eye, the similar viewpoints usually mean there’s a lot of potential for compression, creating smaller, more manageable files for streaming and storage.

Applications of MV-HEVC

Stereoscopic Video (3D Video)

MV-HEVC is particularly useful in the realm of 3D video or stereoscopic content, where two slightly different views (one for each eye) create the stereoscopic effect. By encoding both the left eye and right eye views efficiently in a single stream, MV-HEVC reduces the file size and bitrate compared to other methods. This is crucial for streaming applications like 3D movies or immersive VR experiences where quality and efficiency are key. Other codecs can be used for 3D stereoscopic video as we cover in this blog, but MV-HEVC is more efficient. 

Spatial Video

Another application of MV-HEVC is in spatial video, which is typically used for virtual reality (VR) or augmented reality (AR) content. The Apple Vision Pro is built around the idea of capturing and presenting spatial video, allowing users to immerse themselves in a three-dimensional representation of a scene, combining video and depth information. MV-HEVC support is essential for these types of experiences, reducing massive bitrates of the raw files into something manageable for streaming and real-time immersive experiences. 

Multiview Video

MV-HEVC is also important for multiview video, where multiple views of the same scene are captured from different angles. This could be used in sports broadcasts, where different camera angles are encoded into a single video stream, or for applications that allow users to choose their viewing angle interactively. Depending on your exact use case, this may require multiple decoders or extra processing power that might not be available on all platforms. 

Dolby Vision with MV-HEVC

MV-HEVC is now also compatible with Dolby Vision, a popular High Dynamic Range (HDR) video format that helps ensure content looks as realistic and as true to the creator’s vision as possible. Most of the top-tier premium streaming content these days is being made available in Dolby Vision format, so it makes sense that companies investing in MV-HEVC production pipelines would want to take advantage of Dolby Vision. Dolby Vision Profile 20 extends the potential quality enhancements of Dolby Vision to MV-HEVC and immersive content. 

Apple Vision Pro and beyond

The Apple Vision Pro is pushing the boundaries of immersive media and while they didn’t create the VR headset segment, Apple definitely put their stamp on it. There are several examples over the years of Apple’s influence on the media technology industry, from their decision to not support Flash video to their decision to (finally) support AV1. 

It seems only likely there will be a halo effect for MV-HEVC around the Vision Pro. One early example is the Blackmagic URSA Cine Immersive camera. I expect in 2025 we’ll see more companies venturing into MV-HEVC support from capture to post-production to distribution. 

MV-HEVC video tools

Direct recording with Apple Vision Pro and iPhone

You can record spatial video using MV-HEVC directly on the Apple Vision Pro, iPhone 15 Pro and all iPhone 16 models. The distance between the 2 camera lenses on the Vision Pro seems to provide better results with more depth compared to spatial videos captured on iPhone.

Apple AVFoundation support

Apple also added support to their AVFoundation APIs for converting side-by-side 3D videos into MV-HEVC and spatial videos. You can find more information in their developer documentation here.

Bitmovin VOD encoding beta

Bitmovin’s VOD Encoding now supports MV-HEVC as part of a private beta. If you’re interested in adding MV-HEVC to your transcoding workflows, we’d love to discuss the details with you. You can reply in the Bitmovin Community, comment on this post or get in touch with your Bitmovin contact for more info. 

Conclusion

Thanks in large part to Apple, MV-HEVC is poised to become a key technology in the future of immersive and multiview content. Its ability to efficiently encode multiple views of the same scene, reduce the data required, and maintain high video quality makes it an essential tool for everything from stereoscopic 3D movies to virtual reality experiences on devices like the Apple Vision Pro.

On their other platforms, Apple seems to have signalled a shift toward using the AV1 codec, but AV1 does not currently have multiview support. It will be interesting to see how that situation evolves both within Apple’s products and the wider video ecosystem. While the only certainty is that things will change, unless Apple abandons the Vision Pro, MV-HEVC is likely to be part of the picture for the foreseeable future.

The post Multiview HEVC (MV-HEVC): Powering spatial video experiences and more appeared first on Bitmovin.

Apple’s Worldwide Developer Conference is an annual event used to showcase new software and technologies in the Apple ecosystem. It was created with developers in mind, but sometimes new hardware and devices are announced and its keynote presentations have become must-see events for a much wider audience. There is also usually news about changes and additions to the HTTP Live Streaming (HLS) spec and associated video playback APIs. These HLS updates are often necessary to support new features and capabilities of the announced OS and hardware updates. This post will expand on Apple’s “What’s new in HTTP Live Streaming” document, with additional context for the latest developments that content creators, developers, and streaming services should be aware of.

The lastest HLS updates for 2024

The first draft of the HLS spec (draft-pantos-http-live-streaming) was posted in 2009, then superseded by RFC 8216 in 2017. There are usually draft updates published once or twice per year with significant updates and enhancements. A draft proposal was shared on June 7, that details proposed changes to the spec to be added later this year. Let’s look at some of the highlights below. 

Updated Interstitial attributes

In May 2021, Apple introduced HLS Interstitials to make it easier to create and deliver interstitial content like branding bumpers and mid-roll ads. Now, new attributes have been introduced for Interstitial EXT-X-DATERANGE tags, aimed at enhancing viewer experience and operational flexibility. 

1. X-CONTENT-MAY-VARY: This attribute provides a hint regarding coordinated playback across multiple players. It can be set to “YES” or “NO”, indicating whether all players receive the same interstitial content or not. If X-CONTENT-MAY-VARY is missing, it will be considered to have a value of “YES”.

2. X-TIMELINE-OCCUPIES: Determines if the interstitial should appear as a single point “POINT” or a range “RANGE” on the playback timeline. If X-TIMELINE-OCCUPIES is missing, it will be considered to have a value of “POINT”. “RANGE” is expected to be used for ads in live content.

3. X-TIMELINE-STYLE: Specifies the presentation style of the interstitial—either as a “HIGHLIGHT” separate from the content or as “PRIMARY”, integrated with the main media. If X-TIMELINE-STYLE is missing, it is considered to have a value of “HIGHLIGHT”. The “PRIMARY” value is expected to be used for content like ratings bumpers and post-roll dub cards. 

More detail is available in the WWDC Session “Enhance ad experiences with HLS interstitials“.

Signal enhancements for High Dynamic Range (HDR) and timed metadata

HDR10+

Previously, the specification had not defined how to signal HDR10+ content in a multi-variant HLS playlist. Now you can use the SUPPLEMENTAL-CODECS attribute with the appropriate format, followed by a slash and then the brand (‘cdm4’ for HDR10+). The example Apple provided shows the expected syntax: SUPPLEMENTAL-CODECS=”hvc1.2.20000000.L123.B0/cdm4″. For a long time, HDR10+ was only supported on Samsung and some Panasonic TVs, but in recent years it has been added by other TV brands and dedicated streaming devices like Apple TV 4K and a few Roku models.

Dolby Vision with AV1

Dolby Vision has been the more popular and widespread dynamic HDR format (compared to HDR10+) and now with Apple adding AV1 decoders in their latest generation of processors, they’ve defined how to signal that content within HLS playlists. They are using Dolby Vision Profile 10, which is Dolby’s 10-bit AV1 aware profile. HLS will now support 3 different Dolby Vision profiles: 10, 10.1 and 10.4. Profile 10 is “true” Dolby Vision, 10.1 is their backward compatible version of HDR10 and 10.4 their backward compatible version of Hybrid Log Gamma (HLG). For profiles 10.1 and 10.4, you need to use a SUPPLEMENTAL-CODECS brand attribute and the correct VIDEO-RANGE. For these, 10.1 should use ‘db1p’ and PQ, and 10.4 should use ‘db4h’ and HLG. The full example codec string they provided is: CODECS=”av01.0.13M.10.0.112″,SUPPLEMENTAL-CODECS=”dav1.10.09/db4h”,VIDEO-RANGE=HLG.

If you’re interested in Apple’s overall AV1 Support, you can find more details in this blog post.

Enhanced timed metadata support

HLS now supports multiple concurrent metadata tracks within Fragmented MP4 files, enabling richer media experiences with timed metadata (‘mebx’) tracks. This will enable new opportunities for integrating interactive elements and dynamic content within HLS streams. .

Metrics and logging advancements

The introduction of the AVMetrics API to AVFoundation will allow developers to monitor performance and playback events. This opt-in interface lets you select which subsets of events to monitor and provides detailed insights into media playback, allowing you to optimize streaming experiences further.

More details are available in the AVFoundation documentation and the WWDC 2024 session “Discover media performance metrics in AVFoundation”.

Common Media Client Data (CMCD) standard integration

HLS now supports the CMCD standard, enhancing Quality of Service (QoS) monitoring and delivery optimization through player and CDN interactions. AVPlayer only implemented the preferred mode of transmitting data via HTTP request headers. They have not included support for all of the defined keys and for now is only supported in iOS and tvOS v18 and above. There was no mention of support in Safari. 

Bitmovin and Akamai debuted our joint CMCD solution at NAB 2023. You can learn more in our blog post or check out our demo.

FairPlay content decryption key management

As part of ongoing improvements, HLS is deprecating AVAssetResourceLoader for key loading in favor of AVContentKeySession. AVContentKeySession was first introduced at WWDC 2018 and until now, Apple had been supporting both methods of key loading for content protection in parallel. Using AVContentKeySession promises more flexibility and reliability in content key management, aligning with evolving security and operational requirements. This move means any existing use of AVAssetResourceLoader must be transitioned to AVContentKeySession. 

Conclusion

The recent HLS updates show Apple’s commitment to enhancing media streaming capabilities across diverse platforms and scenarios. For developers and content providers, staying updated with these advancements not only ensures compliance with the latest standards but also unlocks new opportunities to deliver compelling streaming experiences to audiences worldwide. 

If you’re interested in being notified about all of the latest HLS updates or you want to request features or provide feedback, you can subscribe to the IETF hls-interest group.

The post WWDC 2024 HLS Updates for Video Developers appeared first on Bitmovin.

This post was originally published in Nov. 2023. It was updated on Feb 19, 2024, with information about current applications that are available on the Apple Vision Pro.

Apple has always been great at creating hype around its new devices or features and getting millions of people to talk about and use them. This was especially true in September at this past IBC, when many conversations we had at our booth and on other stands included mention of Apple’s big announcement on supporting AV1 for the next version of their iOS devices. With their latest push into the AR/VR space, it’s no different. In early 2024, Apple is expected to release the Apple Vision Pro, which is set to be the next big device streaming services are getting ready to support. However, what does it look like to stream video on these devices, and how will it ensure a high-quality experience for viewers? 

In this blog, we will look at what Apple Vision Pro and visionOS are bringing to the table, the features and industries that can take advantage of it, and how Bitmovin’s Player is focused on supporting the new device.

What is Apple Vision Pro and visionOS?

Apple Vision Pro is an augmented reality (AR) headset announced in June 2023. It features a new three-dimensional interface that users can control with their eyes, hands, and voice. The headset is also powered by their latest operating system, visionOS, explicitly designed for spatial computing. Creating applications for visionOS is pretty straightforward, as Apple has streamlined the development process for all Apple platforms through Xcode. 

Xcode is Apple’s integrated developer environment application for those unfamiliar, enabling you to develop, test, and distribute apps for all Apple platforms. Because of this, our iOS Player team found it to be a very straightforward process to add the new platform to existing projects already deployed on iOS and tvOS. 

Does Apple Vision Pro and visionOS support video streaming?

The short answer is yes. The Apple Vision Pro was made with video as the focus and outputs superior picture and sound quality from its 23 million pixel resolution and dual-driver audio pods that enable an immersive experience for users. It supports up to 4K video, allowing streaming platforms to engage viewers with the highest quality live and on-demand content across browsers and dedicated applications.

Are there active applications streaming on Apple Vision Pro?

As the Apple Vision Pro just launched in the US (scheduled for Europe and other areas later in the year), there is currently a growing number of apps that are available or working to go live on the device. With its immersive augmented reality, every platform streaming video has the ability to provide its users with a way to “step into the screen” for a truly unique viewing experience. Some of the active applications that are offering a great viewer experience on the Apple Vision Pro are:

• NBA (check out a video of their app) and MLB and other sports leagues and broadcasters are bringing viewers onto the field with live games, replays, and stats.
• OTT platforms like Paramount+, Peacock, and others are giving their viewers a way to engage with content in any environment
• Social Media platforms like TikTok have users scrolling their feeds while using other apps at the same time

Which industries and use cases work on the Apple Vision Pro?

Depending on the industry and use case, Apple Vision Pro will be a great way for viewers to stream content. Some of these industries include: 

• Sports
  • Viewers will be able to watch their favorite sporting teams like never before with mixed reality capabilities.
    
• Entertainment and OTT platforms 
  • With 4K video and surround sound capabilities, streaming TV and movies will be a more immersive and personalized experience.

• Esports and gaming
  • Users will be able to follow the latest gaming events closer and stream their favorite gamers.

• Online events
  • With virtual conferences, concerts, and more, users will be able to engage and interact more with content they are interested in.

• Health and Fitness
  • Apple Vision Pro will be used to provide remote workouts, physical therapy, and rehabilitation sessions, making it easier for people to access care from the comfort of their own homes.

• Edtech and eLearning
  • Video learning will be more interactive and immersive, making learning and retaining information more accessible for students.

Does Bitmovin support video playback on Apple Vision Pro?

Yes, we do! As Apple Vision Pro will give users a new viewing experience, it’s vital for streaming platforms to integrate and keep a consistent video quality across all devices and browsers. With Bitmovin’s support for visionOS enabling video playback on Apple Vision Pro, integration is possible today. This makes Bitmovin’s Player the only video player on the market that officially supports visionOS, providing dev teams with a unified development experience across all Apple platforms with our iOS, tvOS, and visionOS SDK.

 Some of the features that are available with our iOS, tvOS, and visionOS SDK. See the whole feature by downloading our one sheet.

To further our development, we recently attended an Apple Vision Pro developer lab in Munich, where we got to test out the Bitmovin Player on physical Apple Vision Pro devices. It was essential to get this hands-on experience as we were able to see firsthand how our Player performed when streaming video up to 4K. Today, Playback capabilities on Vision Pro devices are similar to others in Apple’s ranges. However, Bitmovin will continue to support any additions to functionality or integrations that Apple may introduce to this new device type. You can see our latest SDK updates in our release updates.

The wrap-up

As the industry gets ready for the release of the Apple Vision Pro in 2024, it’s clear there are a lot of streaming platforms working to get to market and reach the new viewer base on the device. Ensuring playback is consistent and in the highest quality across every device will be key for streaming platforms in maintaining viewer retention and engagement. That is why Bitmovin continues to focus on supporting the broadest range of devices in the market and providing a robust integration and feature set to enable our customers to reach their viewers with the best quality of experience possible. 

If you want to know more about Bitmovin’s support for visionOS and playback on Apple Vision Pro, join the conversation in our Bitmovin Community, and if you want to test out our solution suite, sign up for our 30-day free trial.

The post Streaming Video on the Apple Vision Pro: Supporting visionOS with the Bitmovin Player appeared first on Bitmovin.