How Strings Actually Work - with guest Huiqi Zhou

In this video with guest Huiqi Zhou I explain how strings actually work. We go over string encodings, unicode, UTF-8, UTF-16, and more using code examples in Node.js.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : so I wanna tell you about string and
00:06 : coatings and Unicode you said you have
00:08 : no idea what Unicode is right what is
00:12 : Unicode okay so so I made a text file
00:15 : called it word text and I have these
00:19 : different types of characters on it I
00:22 : like pie I mean but name for the hello
00:26 : we're out here
00:27 : okay hello world I like pie to infinity
00:31 : and beyond
00:32 : anyhow rude yeah that's Chinese we're
00:35 : both Chinese speakers so we can
00:37 : understand this other people watching
00:39 : they're out of the loop but we can also
00:42 : put some emojis here what do you think
00:45 : would happen if I wrote an odious
00:47 : program to read in this file and go
00:50 : through the string and print out
00:52 : character by character for now - what do
01:16 : you mean some sort of code that called
01:29 : encoding oh you probably looked at the
01:32 : typescript definition of the read file
01:35 : function
01:36 : so anyway I'm gonna write a note a
01:39 : script that's gonna read in the file and
01:42 : loop through each character of the
01:43 : string and print it out no js' has this
01:46 : FS dot promises
01:48 : API now comes out of the box if you want
01:51 : to use async await with the file system
01:54 : I just eighty is it like this now so
01:57 : that when I say FS I actually mean FS
02:00 : dot promises so this is how I do it now
02:03 : okay so so now that it's the promised
02:06 : version of FS I can await on the read
02:09 : file because it's a promise
02:11 : and then the thing I get back is
02:13 : actually a buffer if you remember we're
02:16 : gonna talk about the buffer also but for
02:18 : now I'm gonna convert it to a string
02:21 : because that's how you learn to read in
02:24 : text files in nodejs you you you convert
02:27 : the buffer to a string yes but for
02:33 : before gives you the it bite array
02:35 : representation of of the content well
02:39 : we're gonna to string it and there's
02:42 : actually some magic to the to string
02:44 : let's do so just gonna loop through in
02:47 : the most classic way possible then I
02:51 : write a for loop like I'm writing see I
02:54 : just use index notation to get each
02:57 : character alternatively you could have
02:59 : done like char at like that but it
03:04 : doesn't really matter which one you use
03:05 : I'm just gonna stick to this one because
03:07 : it looks very low-level okay so I'm
03:13 : gonna print out character by character
03:15 : to the screen I'll print out the index
03:17 : number and the character so the file i
03:21 : made a curried file with a yes I'll run
03:23 : that what just happened promises it this
03:29 : should be promises plural okay okay so
03:33 : did that so notice that we got some of
03:38 : the characters to come out hello world
03:42 : I like pie to infinity and beyond but
03:46 : there's some characters at the end that
03:48 : did not come out and the ones that do
03:51 : not come out are the emojis but but the
03:54 : Chinese characters came out okay and
03:56 : then and they're weird Greek letters
03:58 : also came out okay so that's interesting
04:01 : let's change it to read the bytes like
04:04 : well what would happen if we read the
04:06 : bytes so make another file called it
04:09 : read file byte I'm just gonna make a
04:12 : copy of this file and instead of looping
04:15 : through the string I'm
04:17 : looping through the buffer so I'm gonna
04:18 : get rid of the string at this time I'm
04:21 : just looping through the buffer and when
04:24 : you're looping through the buffer and
04:26 : getting every element of the buffer each
04:29 : value is a byte and that's actually
04:32 : gonna be a number so I'm gonna print out
04:36 : the byte and it's from this file so now
04:40 : I read file bite yes but and you get you
04:46 : got these numbers each from each number
04:50 : is a number between negative 127 I think
04:56 : to positive 128 I believe because a byte
05:03 : is a 8 bit number a bit number is like 0
05:08 : 1 0 0 1 1 0 1 that that's a bite
05:13 : so each byte I'm printing out is a 8
05:16 : digit binary number and if you do 2 to
05:21 : d8 that's how many possible combinations
05:25 : you can have if you want to represent
05:28 : negative numbers then you split the 256
05:32 : in half half of those get to be negative
05:36 : numbers and half of those get to be
05:37 : positive numbers if you're gonna say no
05:40 : I'm only representing positive numbers
05:42 : then you get to represent starting from
05:45 : 0 to 255 right we actually don't need
05:51 : negative numbers but traditionally well
05:54 : the first American character set is
05:57 : called ASCII have you heard of ASCII
05:59 : before yes yeah so ASCII represents 127
06:06 : characters I believe it includes all the
06:09 : other digits from 0 to 9 included
06:13 : uppercase a to z and lowercase a to z
06:16 : the ASCII standard use only 7 bits so it
06:20 : only used seven of these like that that
06:23 : is enough for a ski so we didn't
06:27 : actually need an entire bite there's one
06:29 : bit
06:30 : extra the ascii is not even using so but
06:35 : but later on we standardize on using 8
06:38 : bits for a lot of different types of
06:41 : things so using a byte it's just so
06:43 : commonplace that you're not gonna write
06:46 : a program that uses 8 bits that just
06:49 : never happens anymore you always use
06:52 : multiple of 8 either 8 8 bits or 16 or
06:56 : 32 or 64 right so so it's just a given
07:01 : that you're gonna use at least eight
07:03 : bits at least a byte so so then there's
07:06 : an extra bit is like well what does that
07:08 : do
07:09 : we'll get to that so I hope I'd have the
07:13 : Internet
07:14 : because I kind of wanted to show you
07:16 : house key table dot com okay so yeah
07:19 : this is this is all of ASCII right it
07:22 : starts from code 0 to code 127 which
07:26 : which is the Delete key and you know the
07:30 : Elva numerals they're in here somewhere
07:32 : and there's the space and exclamation
07:34 : point everything that's on your keyboard
07:36 : is on here somewhere right but obviously
07:39 : that's a problem because
07:40 : where are our pies and infinities and
07:43 : Chinese characters and our emojis those
07:45 : are not there I have a number right a
07:48 : byte is a number how can I get the
07:51 : character representation which that
07:53 : number is based on in JavaScript is
07:55 : string dot from chart who'd that this
08:00 : byte is some ASCII code right this byte
08:05 : number is some ASCII code and from the
08:08 : ASCII code I can get JavaScript to tell
08:11 : me give me a character based on that
08:14 : ASCII code so we can do this yeah let's
08:18 : do that
08:18 : so print out both the code and the
08:21 : character and you'll see that yeah the
08:24 : hello world that comes out fine I like
08:27 : comes out fine the PI did not come out
08:32 : to to infinity did not come out and
08:36 : Beyond and the
08:37 : Chinese characters did not come out and
08:39 : the emojis also did not come out like
08:48 : why do we have weird characters here yes
08:51 : like there's a hence a fraction 1 1 / 2
08:57 : fraction and a 1 / 4 fraction that's
09:00 : weird so it turns out that later on
09:04 : remember we said there's 8 bits right
09:07 : but ASCII only uses 7 of the bits later
09:11 : on people said wow that's kind of you
09:14 : know I can use I can do a lot with that
09:16 : extra bit I can represent another 128
09:23 : characters with that extra bit so don't
09:26 : they they just said they brought in an
09:29 : extra 128 characters they caught that
09:32 : extended ASCII and then and then they so
09:36 : they added extra characters like you
09:40 : know like this like this a with the two
09:43 : dots on the top and this is this yen or
09:45 : something I don't know and in this
09:48 : fraction they brought in extra
09:50 : characters because there's a whole extra
09:53 : bit that way they can use to get 128
09:56 : more combinations so so this is
09:58 : basically falling back to that when when
10:01 : we're told the terminal display this
10:03 : ASCII code and then terminal is a well
10:06 : this is out of the ASCII range so it
10:08 : uses the extended ASCII but obviously
10:11 : the extended ASCII still cannot
10:14 : represent Chinese characters and emojis
10:18 : the like there's not enough combinations
10:21 : to represent all that many different
10:24 : things right because those are way out
10:26 : of range if you wanted to represent all
10:29 : the Chinese characters you need a lot of
10:31 : different combinations to do it so this
10:35 : is not enough so what is the solution
10:37 : the solution that we thought was going
10:41 : to be fine at some point
10:43 : this was in the 90s um at some point I
10:46 : think and I remember when I was college
10:48 : people were talking about
10:51 : is we'll just extend it to 16-bits if we
11:00 : have 16 then we can represent a lot more
11:04 : we can actually represent two to D 16
11:06 : which is 65,536 different possible
11:12 : characters in the character set exactly
11:16 : that's not enough but back in the 90s
11:19 : people thought that was enough like this
11:22 : is all we need
11:23 : we're good now we have a 65 K some
11:27 : characters and we're enough so the
11:29 : Unicode is a table that has all
11:35 : characters that we might ever want so
11:39 : there's a group of people that
11:41 : standardized it and created the Unicode
11:45 : which is a mapping of characters to
11:48 : numbers how many characters are in
11:52 : Unicode there's more than a million
11:55 : characters in the Unicode standard which
11:57 : is obviously more than 65,000 so you're
12:01 : like how can 65,000 work well people
12:04 : thought that would work but it didn't
12:06 : work so this is the Unicode table you
12:10 : can see some of the letters that are in
12:13 : this extended ASCII set that one over
12:15 : two and the one over four that this is
12:19 : that's still within the 256 range right
12:23 : you can go down to the Chinese character
12:26 : ranges and you can see the Chinese
12:28 : characters are within the 65,000 range
12:33 : so if you have 65,000 something
12:35 : characters you can actually represent
12:38 : Chinese characters and a lot of
12:40 : different languages in the world
12:52 : nice yeah maybe I can type it you know a
12:56 : character in their name that's weird it
12:59 : doesn't try to use opinion so it's
13:03 : so it's not on this page go down I
13:09 : believe it might not be on this there
13:13 : are some Chinese characters that are
13:16 : after the 65k range that because they
13:20 : had to supplement it they realized
13:23 : there's some they missed and then later
13:25 : on they added to the standard printing
13:28 : the bytes divides them into 8-bit chunks
13:32 : right but the first program that I wrote
13:35 : which iterated through the characters
13:37 : guess what what is the chunk size there
13:41 : it's 16 bits if you loop through the
13:45 : string based on string that length and
13:47 : you use the indexing to get at each
13:50 : character then the character is
13:54 : represented by 16 bits that's at one
13:57 : point people who made program languages
14:00 : thought 16 bits is all we needed to
14:05 : represent all possible characters and
14:07 : they designed a programming language
14:09 : like Java JavaScript they basically said
14:13 : I'm gonna make this thing called the
14:14 : wide character previously characters
14:17 : were 8 bits
14:18 : we're gonna have wide characters in Java
14:21 : and a character is 16 bits and then of
14:24 : course later on they realize actually
14:27 : even 16 bits is not enough so how are we
14:30 : gonna iterate through the emojis and any
14:33 : character that's outside of the 65k
14:36 : limit well let me make another version
14:38 : of this file interestingly they added a
14:42 : new feature of JavaScript such that if
14:45 : you iterate through the string
14:47 : differently everything just works so if
14:52 : you if instead of using the classical
14:55 : for loop you switched it to 4 what am i
14:59 : using var for I should use let or cons
15:03 : always obviously if you switch to the
15:07 : for off loop to iterate to the string
15:11 : and then you print out the character
15:15 : everything just works amazingly why
15:19 : see our Unicode characters are coming
15:22 : out correct now and the Chinese
15:25 : character basically all of the
15:26 : characters are coming out correctly all
15:29 : you have to do is change the for-loop to
15:32 : a four of loop what why is that well so
15:36 : at some point drivers get introduced a
15:38 : thing called iterators do you know about
15:41 : iterators so iterator is like symbol
15:47 : iterator symbol which you can use as a
15:51 : key you've seen this weird thing right
16:02 : but once you have this iterator thing
16:05 : then you can keep calling iterator dot
16:07 : next right so we'll just call it like a
16:12 : few times and print it out but basically
16:15 : this is the lower low level way that
16:18 : this stuff works
16:19 : but the for off loop is internally using
16:22 : this iterator so yeah that worked so if
16:31 : if you do it this low-level way you call
16:34 : this symbol iterator method and then you
16:39 : get the iterator and you call dot Nix
16:42 : Nix Nix until the done becomes true so
16:46 : you would write a while loop like this
16:48 : like wild true result is iterator that
16:56 : next and then you say if result that
16:59 : done I'm gonna break out of the loop
17:02 : right and then you print the result here
17:06 : so if you have done this it'll loop
17:10 : through based on this iterator and print
17:13 : out all the characters so that works and
17:16 : yeah there's this wrapper object that
17:19 : has the value plus the done value right
17:21 : I can just do that value if I don't want
17:24 : to see all this stuff but internally
17:27 : that's what basically what for
17:29 : of does four of four off does that so
17:32 : basically they define a special string
17:35 : iterator that understands Unicode it
17:38 : will know if this character requires
17:41 : more than two bytes to represent based
17:44 : on the encoding of the character and and
17:48 : then if it does require more bytes for
17:51 : for those multibyte characters it'll
17:55 : actually be more than 1:1 in length
17:58 : so I can show you that by doing char
18:00 : that links because char is also a string
18:03 : right I don't need this guy anymore yeah
18:09 : so so these things have length 1
18:12 : but if you get down to the emojis emojis
18:15 : have a length of 2 the Chinese
18:18 : characters still have a length of 1
18:19 : because they fit into that 65,000
18:22 : boundary so emojis came really really
18:26 : late so they have to come after the
18:29 : 65,000 Kay boundary or very unusual
18:33 : Chinese characters also come after that
18:40 : [Music]
18:43 : multibyte quick great question so I
18:49 : wrote I wrote some other programs I'm
18:53 : gonna run this one called
18:54 : read file unified and I'm gonna post
18:57 : this code on github later so in case you
19:00 : or other people watching wanna see them
19:03 : so I wrote several versions of this read
19:07 : file program the run one that will show
19:10 : you the binary encoding in all three
19:13 : sizes in 8-bit sizes in 16-bit sizes and
19:18 : in 32-bit sizes all together
19:20 : side-by-side okay let me fix the syntax
19:28 : error I was trying to write this
19:30 : function but I and then I get distracted
19:32 : and went to do something else probably
19:34 : okay I'm actually because there's so
19:38 : much input I'm gonna take away some of
19:41 : the
19:42 : characters so that you can see
19:44 : everything in one screen so maybe I'm
19:45 : gonna put I'm just gonna put holo on one
19:50 : line and then PI and infinity and then
19:55 : let's put the Chinese characters and
19:58 : then the emojis here okay okay
20:05 : so now I can see everything in one
20:06 : screen so if we I don't know why I have
20:11 : two 8-bit in the middle but the 8-bit
20:14 : representation is that is this center
20:18 : column the 16-bit representation is this
20:21 : column to the right and the 32-bit
20:23 : representation is the column over here
20:26 : to the left this decimal and hex codes
20:29 : those are the numbers that represents
20:33 : the character number within the unicode
20:38 : system because the unicode system is
20:41 : just a doesn't care about how many bits
20:45 : you're using to represent the number the
20:47 : unicode system is just like a character
20:51 : zero is the number 48 it's just based on
20:55 : a straight number line but but if once
20:59 : you encode that number which might be
21:01 : very very big like a million if you need
21:04 : to represent the number a million you're
21:06 : gonna need more than eight bits or 16
21:09 : bits you're gonna need 32 bits to
21:11 : represent a million so so how do you
21:15 : represent a very very large number in 8
21:18 : bits well the first byte it's gonna have
21:21 : to tell you that there's actually more
21:22 : bytes coming behind me right and the way
21:29 : you you tell the program that's reading
21:32 : you that there's more bytes coming after
21:35 : you is you flip this first bit to 1 you
21:38 : see how the other ones that are only
21:41 : that you require only one byte have a 0
21:45 : at the beginning right so these ones are
21:48 : the ones that fit into the ASCII set
21:51 : they only require one byte
21:53 : represent but if there's multiple bites
21:56 : that are needed the first bit is gonna
21:59 : be one that tells you there's more bites
22:03 : how many bites are coming after me
22:05 : that's based on how many ones come after
22:09 : the first one so is that this one
22:13 : there's one one that comes after this
22:15 : one so there that means there's one more
22:18 : bite coming after me so this one means
22:23 : there's more more bites and this one is
22:27 : there's one more bite and then the next
22:31 : one is zero that means there's no more
22:34 : that there's just one more bite but for
22:36 : the one that has two more bites it's a
22:38 : one one one see that is true
22:42 : there's two more ones after the first
22:43 : one that means two more bites are coming
22:46 : and then for the one that needs four
22:49 : bites it's one one one one the first one
22:52 : is the signal bit and then we have three
22:54 : more ones to tell you that there's three
22:57 : more bites that are coming after ya so
23:03 : so it's a variable-length characters
23:05 : basically and if you're writing a
23:08 : program that needs to interpret this
23:10 : correctly you need to write logic to
23:12 : handle this encoding and and that's why
23:16 : this thing this concept called encoding
23:19 : is important because you can't have you
23:22 : can't interpret a file that contains
23:25 : text in it unless you know what encoding
23:28 : it was encoded in right this is it and
23:32 : encoded based on utf-8 or was it encoded
23:36 : based on utf-16 or utf-32 so so with UT
23:43 : you do 32 the problem does not exist
23:47 : because 32-bit is enough to represent
23:50 : more than a million actually a lot more
23:54 : than a million so if you use 32 bits
23:57 : you'll probably be able to represent all
23:59 : possible characters in another century
24:05 : yeah but but but the problem with third
24:09 : utf-32 is look how wasteful we are we
24:12 : have all these zeros like we're using
24:15 : all of these memory extra memory and
24:18 : we're just filling up with useless
24:19 : information they're just zeros so if
24:23 : you're gonna write your webpage for
24:26 : example and you're gonna encode it in
24:28 : utf-32 you're the transfer of your
24:31 : webpage it's gonna take a lot more
24:34 : bandwidth and also a lot more disk space
24:37 : to store the file on disk right and also
24:40 : a lot more memory for the web server
24:42 : that has to load the stuff into its
24:44 : memory maybe well if it's a streaming
24:47 : web server not not as big a problem but
24:50 : but it's still there so that's why
24:54 : utf-32 is not popular and utf-8 is
24:58 : actually the most popular one that is
25:02 : used for file encodings when you write
25:04 : code and you save the code as a dot
25:06 : JavaScript file ts file it's usually
25:09 : being saved as utf-8 because it it's
25:12 : saves space right the files are smaller
25:15 : you transfer it the transfer is smaller
25:18 : faster etc however utf-16 is very
25:25 : popular in programming languages as in
25:28 : when a programming language reads in
25:30 : your file it will convert but usually
25:34 : the file is in utf-8 but in memory it'll
25:37 : convert it into utf-16 because the
25:40 : languages thought that was a good idea
25:42 : at the time so JavaScript does this Java
25:48 : does this even even though JavaScript
25:50 : when it reads a file it'll default to
25:52 : using utf-8 to interpret the file when
25:56 : it gets into memory it has translated it
25:58 : into utf-16 so so when you have a string
26:04 : in JavaScript it looks like this and I
26:08 : won't explain how it's it's a similar
26:11 : concept there's some some bits that's
26:14 : signo the fact that there's one more
26:15 : chunk coming after me basically with but
26:19 : yeah the sixteen but is able to
26:22 : represent a lot more and you're only
26:23 : ever gonna have one more chunk coming
26:26 : after you because one more chunk gives
26:29 : you 32 and that can represent enough
26:31 : like all the characters in Unicode so
26:34 : that's how that's kind of how único
26:36 : works I honestly I didn't know a lot of
26:41 : this until until this last week when I
26:44 : went reading up about it
26:45 : and if you're wondering about nodejs the
26:49 : the buffer dot two string method
26:50 : actually has an encoding parameter
26:53 : that's optional and it defaults to utf-8
26:58 : so usually because usually text files
27:03 : are encoded in utf-8 so that's not a
27:06 : problem for you you never had to put
27:08 : that in but in there very very rare case
27:11 : if the words are encoded in a different
27:14 : encoding like utf-16 little-endian then
27:18 : you're gonna have to specify that in
27:20 : order to parse it correctly otherwise
27:22 : everything looks like garbage if you get
27:24 : it wrong yeah thanks thanks for joining
27:43 : me on this video
27:44 : cool thanks for sharing those very
27:47 : interesting