Patrik Nusszer's programming blog

Youtube deciphered

;

Introduction in abstract

There was a time when youtube downloaders were flourishing. You just googled one, and you were sure that the first link will lead you to a tool that will let you download you favourite band's latest music video. But you know that world is based on money. There's money in everything. In my website (ads), and Youtube too. When you just simply download a video from youtube, you just do not watch their advertisements and obviously you have no chance to click it, for which Youtube, let's call it Google, its respective owner, receives money. So they invented an easy crypto system to protect Vevo videos. That was the time when unexpectedly all downloaders went wrong. But this is only true for Vevo videos. Other videos can still be downloaded by using the old school method: either downloading the page of the video and extracting the stream URLs. Or, the second option, is to download through Youtube's API. However skilled you are, I'm going to lead you through how to practically download Vevo videos.

The 2 approaches

So there 2 ways to retrieve stream information, the first is the simplest one.

Downloading the video page

which has a standard URL that looks like: https://www.youtube.com/watch?v=VIDEO_ID_HERE
In this case, the stream information is embedded inside the HTML code, and can be extracted from it using simple regular expressions.
At the time writing, my youtube downloader applications use:
- "ytplayer\.config\s*?=\s*?(\{[\u0000-\uFFFF]*?})\s*?;\s*?ytplayer\.load".

The first (and only one) capturing group contains nothing but the pure JSON data of the streams.

The other method was more widely used by Youtube downloaders.

get_video_info.php

which is not the way I suggest you to use. You are guessing what parameters does it take. Actually that part makes it hard. Because every parameter is easy to give: "video_id=YOUR_VIDEO_ID", "asv=3", "eurl=https://www.youtube.com/watch?v=YOUR_VIDEO_ID", "el=info". So all of them are quite straightforward, except one.

The "sts" param

This parameter requires you to give the id of a crypto type. Yes. To make programmers a harder job, Youtube varies these crypto algorithms, and old ones sometimes get outdated. What this means is that it is not suggested to statically program one, and always send the same id as the sts parameter as it may not work by tomorrow. Actually, my downloaders, when using the API solution, try to find the ONE THAT WORKS. And the id, or the algorithm can only be found in one place.

The player and page

Let me tell you this story: when you download the page, and apply the regex I gave, and extract the JSON data, you get 2 most important data. But are both of them are of importance? Let's continue: these 3 data are:

sts value, the obfuscated player, and streams

These all can be found inside the JSON inside the HTML page of the video. Now go deeper into which is really of importance. In the JSON data, you get the URL of the HTML5 youtube player's obfuscated source code, which in itself contains the method that deciphers the signatures of the stream URLs that arrived in association with it through the JSON data. So to make things clear, the obfuscated player already contains THE ALGORITHM that we need.
In itself, this is perfectly enough for us to decrypt the signatures.
So what about the sts? I tell you: the sts value will order the API to encrypt the singatures according to the same algorithm that is found inside the obfuscated player.

Conclusion of page and API

The JSON data found inside the HTML page of the vidoe provides us all the information that we need to decipher the signatures.
The sts value, that, according to my knowledge can only be obtained through the page method, is quite useless, as you need to use the JSON found in the page to get it, you can'T skip this stepm, so then why to use the API when everything you need is inside the page?
Also, from all this we can infer that when using the API solution, Youtube assumes that you know the algorithm associated with the sts, but it is by no means so. So when you use the api, you either do the:
- backwards solution: statically program into your app one algorithm, and always apply the same sts. Then wait until Youtube outdates it and your app needs to be updated to a newer algorithm.
- download the page, from which you can retrieve the sts, algorithm, and stream URLs. The last two would be perfectly enough for you, but as you chose to use the API solution, use the given sts and it will return similar stream information with URL signatures encrypted by the same algorithm that you hav already retrieved by the page.
All in all, you had better going with the page solution.
In the followings, we are going to discuss how to get the algorithm from the obfuscated palyer source, and the differences between the API and page stream informations.

How to decide whether it is ciphered or not?

So ok, I trust you that you know about your own video whether it is Vevo or not. (Encrypted or not.) But when it comes to build an app for other users who except your app to make this decision by a simple URL, things are a bit more complicated, and this depends on whether you process API streams or page streams

Ciphered or not?: Page

So we arrived to the point where you obtained the JSON object from the page HTML. How do you access the streams? There are 2 types of streams:
- adaptive_fmts: audiovisual
- url_encoded_fmt_stream_map: either video or audio

At the time writing, the JSON "path" to these are:
- string adaptive_fmts = (string)jsonConfig["args"]["adaptive_fmts"];
- string url_encoded_fmt_stream_map = (string)jsonConfig["args"]["url_encoded_fmt_stream_map"];

Then, you need to create 2 arrays, each will contain the URLs of the 2 types. You need to split the just mentioned strings by character ',' which will separate the URLs from each other. Then you should set up 2 "for" loops for each of the arrays of URLs of the 2 types of streams.
Then, when you process 1 stream, you are actually processing its URL. You only need to do anything to it in case it is ciphered. What you need to do is, to split up the URL by the character '&'. Then, you get the URL parameters, each consists of a key and a value. You need to split each parameter by character '='.
Then, if a key "s" exists it is ciphered, and its value is the encrypted signature, otherwise not.

Ciphered or not?: API

The case about API is that the information it returns differs from the data inside the page.
First off, the data returned by it is NOT in JSON format.
This is how you create an organized array of it:
You need to split the whole data by character '&', then getting the parameters, you split each result by character '=', by which you retrieve keys and values, from which you can create a JSON object.
Next, at the time writing, the JSON "path" to the streams are:
- var adaptive_fmts = jsonConfig["adaptive_fmts"];
- var url_encoded_fmt_stream_map = jsonConfig["url_encoded_fmt_stream_map"];

Then, you need to create 2 arrays, each will contain the URLs of the 2 types. You need to split the just mentioned strings by character ',' which will separate the URLs from each other. Then you should set up 2 "for" loops for each of the arrays of URLs of the 2 types of streams.
Then, when you process 1 stream, you are actually processing its URL.
Having these done, the way you decide whether the URLs are ciphered or not is:
- string isCiphered = jsonConfig["use_cipher_signature"];

The result is either "True" or "False". Decipher or not according, you are good to go.
The only thing remaining is to process the obfuscated player source, which is hardest of all...

If ciphered: processing the obfuscated player source

This is a hard topic, as even though using regular expressions you can easily find the algorithm, Youtube sometimes updates the source code and they stop working. But it is not something that you should make back away from programming: it can take months when Youtube changes the source. It is worth much more than using the statically typed and update requiring API method. So:

Function name regex

So at the time writing, you can get the decryption function name by this regex:
- ".\.sig\?.\.set\("signature",f.sig\):.\.s&&.\.set\(\"signature\",(.*?)\("
The name is in the first (and only) capturing group. After this, you'll need to escape all characters that have a special regex meaning as the function name will be pasted into another regex.
An older regex for the function name is:
- "\.sig\|\|([$_a-zA-Z0-9]+)\(";

The function and its parameter

Having the function name, we need to retrieve the body of the function.
There are multiple regexes, sometimes one work while the other does not... You just need to verify whether the regex succeeded and if not try an other one.
- First regex: "function\s" + THE_ESCAPED_FUNCTION_NAME + "\(([\w$]+)\)\{[^\}]*\};"
- Second try: "var\s" + THE_ESCAPED_FUNCTION_NAME + "=function\(([\w$]+)\)\{[^\}]*?\};"
- Third and last try is: THE_ESCAPED_FUNCTION_NAME + "=function\((\w+?)\)\{[\u0000-\uFFFF]*?\}"

The function is found in the 0th capturing group.
The signature object that is passed to the function as a parameter is in the first capturing group.
You may need to escape the signature object / parameter as it is going to be used in another regex.

Getting WRAPPER.WRAPPED_FUNCTION(SIGNATURE, ACTION_PARAM)

In some earlies times, the function we have just parsed used to contain directly the actions that must be done on the parameter / signature in order to be decrypted.
But things changed, there are no longer such example. The actions got wrapped inside other functions with obfuscated names all contained int the wrapper object. But even though it stayed constantly like this, we need to verify whether this wrapper object exist with a regex:
- "([\w$]+)\.[\w$]+\(" + THE_SIGNATURE + "[^\)]*\)"

If the first capturing group is not equal to null, we can continue. Don't forget, for performance reasons, it is enough to evaluate the regex match on the function, not on the whole source code.
This regex matches the title's form of "WRAPPER.WRAPPED_FUNCTION(SIGNATURE, ACTION_PARAM)".

Processing WRAPPER.WRAPPED_FUNCTION(SIGNATURE, ACTION_PARAM)

So let's purely parse the form without capturing anything with this regex:
- "[\w$]+\.[\w$]+\(" + THE_SIGNATURE + "[^)]*\)

With this, you should have a list of these forms. In JS, use global flag, in C#, use regex.Matches(THE_FUNCTION).
Now set up a "for" loop for each of the forms. Now you are going to parse its essential elements.
When processing one single form of "WRAPPER.WRAPPED_FUNCTION(SIGNATURE, ACTION_PARAM)", apply the following regex:
- "([\w$]+)\.([\w$]+)\([\w$]+,?([\w$]+)?\)"

The second capturing group shows the name of the WRAPPED_FUNCTION, the third capturing group shows the ACTION_PARAM.
Create a list of these, as they belong together.

Looking up the WRAPPED_FUNCTION

Now, running our "for" loop, we are going to look up the WRAPPED_FUNCTION, which contains the REAL_FUNCTION, whose parameter is equivalent to the one of WRAPPED_FUNCTION's ACTION_PARAM, as passed on to it.
This regex is:
- "WRAPPED_FUNCTION_NAME:(function\\([$\\w,]+\\){[^}]+})"

The first capturing group shows the body of the WRAPPED_FUNCTION.
Now knowing that only 4 real actions / REAL_FUNCTIONs exist - slice, splice, swap, reverse - we only need to find which of the actions' name is contained inside the WRAPPED_FUNCTION
Each of them takes 1 parameter, and it is the corresponding ACTION_PARAM.
Now you can create a KeyValue array aka dictionary that contains The REAL_FUNCTION and the corresponding ACTION_PARAM.
It is essential to mention that you are executing this lookup regex on the whole source, if it is not obvius.
Also note that:
1) Reverse has no parameter.
2) "Swap" is a commonly accepted name. In fact all the other 3 functions are pure JS functions, "swap" describes a case if none of the others' names weren't found. If so, what the function does is implemented, and you should copy and paste and port the code to your language. The good thing is that if a name is not found, the implementation is always the same: if there were more we should search for implementations: function bodies.

The execution of the list of REAL_FUNCTION, ACTION_PARAM

I'm going to publish here the source of the 4 REAL_FUNCTIONs in C#, and you only need to perform the list of the right order of the REAL_FUNCTIONs and corresponding ACTION_PARAMs.
Here they are:


private static char[] slice(char[] signature, int from)
{
string g = String.Join("", signature);
g = g.Substring(from);
return g.ToCharArray();
}

private static char[] splice(char[] signature, int count)
{
List<char> tmplst = signature.ToList();
tmplst.RemoveRange(0, count);
string g = String.Join("", tmplst);
return g.ToCharArray();
}

private static char[] swap(char[] signature, int at)
{
char c = signature[0];
signature[0] = signature[at % signature.Length];
signature[at] = c;
return signature;
}

private static char[] reverse(char[] signature)
{
List<char> tmplst = signature.ToList();
tmplst.Reverse();
return tmplst.ToArray();
}