In the battle for the Game of Attention, content is the two-edged sword and user experience is the horse that leads it to battle.The streaming wars are in full blast with Netflix positioned for continual dominance, armed with over 50,000 individual titles, 200million subscribers in 200+ countries, even more impressive is the underlying technology powering this growth. In this series, my goal is to scratch more than the surface of how Netflix operates, digging deeper into the technicalities of the end-to-end processes involved in delivering content at scale. In the maiden edition of this series, I focus on the Netflix content onboarding system and open connect. This diagram shows an overview of the system we will be discussing in this article.
Content Content Content
Netflix has a combined library of over 50,000 titles, supporting over 2200 devices, each device with its resolution and network speed. For them to be able to serve that many devices at different Network speeds they need to have the original video in different formats. Netflix receives the video from the production houses in the best possible format. The thing is the videos from the production houses are large, very large. For a commercial blu ray 2hr movie, you're looking at 15-25Gb. Serving this to users in that format will consume data and bandwidth, So Netflix performs a series of preprocessing on the original videos to convert them into different file formats. These preprocessing are referred to as Encoding and Transcoding.
Encoding is the process of compressing video and audio files to be compatible with a single target device. Transcoding, on the other hand, allows for already encoded data to be converted to another encoding format.(MP4,WLM,MOV,MPEG-4) This process is particularly useful when users use multiple target devices, such as different mobile phones and web browsers, that do not all support the same native formats or have limited storage capacity. The reasons for this preprocessing are fairly simple.
- Reduce file size.
- Reduce buffering for streaming video.
- Change resolution or aspect ratio.
- Change audio format or quality.
- Convert obsolete files to modern formats.
- Make a video compatible with a certain device (computer, tablet, smartphone, smartTV, legacy devices).
- Make a video compatible with certain software or service.
Compressing a 25Gb movie will take a lot of time, to solve this problem. Netflix breaks the original video into different smaller chunks and using parallel workers in AWS EC2, it performs encoding and transcoding on these chunks converting them into different formats (MP4, MOV, etc) across different resolutions(4k, 1080p, and more).
Netflix also creates multiple replicas of the same video chunk to cater to different network speeds. About 1300 replicas of a video chunk.
Example
- High Quality ------4K,1080p,720p,360p
- Medium Quality ------4K,1080p,720p,360p
- Low Quality ------4K,1080p,720p,360p
Netflix stores all these processed video data on Amazon S3.Which is a highly scalable and available storage platform for storing static data. Each video file is stored in chunks of scenes.
Netflix uses AWS for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more—hundreds of functions that in total use more than 100,000 server instances on AWS.https://aws.amazon.com/solutions/case-studies/netflix
By default, most chunks are 10 seconds of video. Each time you skip to different parts of a movie you are querying Netflix's Playback API for different video chunks. You usually have one chunk actively playing, one chunk ready to play when the current chunk is done, and one chunk downloaded. This is done to deliver a seamless watch experience to the user and maximize the best possible speed at that moment.
The speed at which these chunks arrive determines the bit rate(bit rate refers to the number of bits used per second) of the following chunks.
If the first one takes longer than the chunk’s playback duration, it grabs the next lower bitrate chunk for the following one. If it comes faster, it will go for a higher resolution chunk if one is available. That's why sometimes your video resolution can be grainy at first and then adjust to a better definition.
Netflix has users in over 200 countries. If a user in Nigeria wants to watch a movie on an amazon instance hosted in America. The internet service providers will need to travel to America to access those servers which will take time and bandwidth. In an era where fractional delays in serving content can lead to declines in revenue, it is paramount that users get access to the content fast. Netflix solved this problem by creating mini servers inside ISPs and IXP(internet exchange points) known as open connect. These boxes are capable of storing 280 terabytes of data. After the videos have been compressed and transcoded on Amazon S3 they are then transferred to these open connect boxes during an off-peak period (Let's say 4 am). When the user requests for a video instead of hitting Netflix servers directly in the US. It hits the open connect boxes. This ensures less bandwidth, faster playtime, and ultimately a better user experience. Movies can also be localized depending on the region. You could store different movies on the open connect appliances in Nigeria and Brazil. About 90% of Netflix content is served this way.
Open Connect
A CDN is a geographically distributed group of services that work together to provide fast delivery of internet content. Open Connect is Netflix’s custom global content delivery network (CDN). Everything that happens after you hit play on a video is handled by Open Connect.
- Open Connect stores Netflix video in different locations throughout the world. When you press play the video streams from Open Connect, into your device.
- Netflix has been in partnership with Internet Service Providers (ISPs) and Internet Exchange Points (IXs or IXPs) around the world to deploy specialized devices called Open Connect Appliances (OCAs) inside their network.
Open Connect Design Image from Netflix - These servers periodically report health metrics optimal routes they learned from IXP/ISP networks and what videos they store on their SSD disks to Open Connect Control Plane services on AWS.
- When new video files have been transcoded successfully and stored on AWS S3, the control plane services on AWS will transfer these files to OCAs servers on IXP sites. These OCAs servers will apply cache fill to transfer these files to OCAs servers on ISP's sites under their sub-networks.
- When an OCA server has successfully stored the video files, it will be able to start the peer fill to copy these files to other OCAs servers within the same site if needed.
- Between 2 different sites which can see each other's IP address, the OCAs can apply the tier fill process instead of a regular cache fill.
Open Connect Transfering content from Netflix
Conclusion
In summary:
- Netflix performs compression and transcoding of the original video file.
- The file is split into chunks using parallel workers in AWS for faster processing.
- Each chunk is subdivided across several resolutions and internet speeds.
- These chunks are stored on Amazon S3.
- During the off-peak period. The files are transferred to open connect boxes, which is Netflix's custom content delivery network spread out across the world.
- These boxes are capable of communicating and sharing content between themselves.
In the next part of the series, I will be explaining Netflix's core backend architecture. Follow me here and across my social media for more content like this Twitter Linkedin