How to Make a Parser with Nearley.js - Part 4

In this episode we’ll demonstrate how to parse a string literal in JSON properly, continuing our series on building parsers using Nearley.js.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : hello welcome to another drive-by code
00:02 : session we're gonna continue on with the
00:06 : the implementation of the JSON parser
00:10 : this time we're going to do a proper
00:15 : implementation of parsing quoted strings
00:19 : and currently we do have a version that
00:23 : works just in an incomplete way so in
00:29 : terms of parsing strings so this is our
00:31 : definition for really at the present the
00:36 : definition for how we purse strings are
00:41 : these three rules so we got string
00:43 : literals this is a literal double quote
00:46 : and that's a literal double quote and
00:49 : between that is a bunch of characters so
00:52 : you might imagine a string like this
00:56 : it's got a double quote literal on the
00:59 : left and to start the string and at the
01:03 : end to close off the string
01:05 : in the middle there's any number of
01:07 : arbitrary characters what is the
01:11 : definition of characters well a
01:12 : character is a characters is one or more
01:18 : characters in essentially and it does
01:22 : this by using this recursion trick that
01:26 : we talked about if you're not familiar
01:29 : with the recursion trick you can you can
01:32 : look at that in more detail in going
01:36 : back and watching the first episodes but
01:38 : basically what this is saying is a
01:40 : characters is either one character or a
01:44 : character followed by more characters
01:46 : which kind of recursive back to itself
01:49 : and this trick allows you to have a
01:51 : string of characters basically and then
01:54 : finally what is the definition of a
01:56 : character this guy here that is a
02:00 : character class matcher
02:05 : match her that says this carrot has a
02:08 : special meaning that means not so this
02:13 : is matching anything that is not the
02:17 : double quote this is an escape slash for
02:20 : the double quote character so basically
02:21 : this whole thing means anything nah that
02:27 : is not a double quote character so but
02:31 : we actually need to be able to allow the
02:34 : double quote character don't we because
02:37 : we can potentially have strings which
02:41 : have double quotes in them and in
02:45 : JavaScript how do you do that well in
02:47 : JavaScript you can do use single quotes
02:49 : to to delimit the string but that's
02:53 : actually not allowed in the JSON
02:55 : specification so this is allowed in
02:57 : JavaScript but not JSON in JSON the only
03:00 : way to do double quote is if you escape
03:03 : it with a slash like that and you want
03:08 : to do another one you need to do another
03:10 : slash and it turns out there's more than
03:13 : just the double quotes that can be
03:15 : escaped and all that is actually
03:20 : described here in the date in the JSON
03:23 : dot or which is the official website for
03:27 : JSON and the official website for JSON
03:30 : actually has the entire grammar of JSON
03:34 : listed out on the right-hand side here
03:36 : you can actually look at exactly how the
03:40 : string is specified and as you can see
03:44 : it's in the grammar and the grammar
03:46 : format is not that far off from the
03:49 : grammar format that we're working with
03:51 : in nearly so the plan for this episode
03:55 : is simply to sort of rewrite these
03:59 : grammar for for matching the strings in
04:03 : our parser here over here so that's what
04:07 : we're gonna do again why cat did all of
04:11 : the code I'm just replaying what he did
04:13 : and explaining it white kid is sort of
04:16 : gonna take these rules
04:19 : by line almost and translate it into
04:22 : this near lease syntax here but
04:25 : essentially what this guy is saying is a
04:28 : character can be anything this guy this
04:33 : this is a character rank it's tip done
04:37 : at a different format but essentially it
04:39 : does meant to say any any character at
04:42 : all my handwriting
04:46 : any character at all and this is - the
04:49 : double quote or the single slash why do
04:52 : we know why we - the double quote but I
04:55 : do we - the single slash there's a good
04:59 : reason for that is because we want to
05:01 : match the single slash in the second
05:05 : rule in a separate rule which in that
05:09 : but in the second rule the single slash
05:11 : starts this escape sequence and we don't
05:15 : want the single slash to be matched by
05:18 : this first rule which will potentially
05:22 : cause the single slash to escape the
05:27 : second rule so when we do see the single
05:29 : slash we'd won that for sure go through
05:32 : the second rule yeah so that's the plan
05:37 : so let's try this so first we're gonna
05:40 : have a single character be any character
05:45 : which is not that's what the the carrot
05:49 : the disc err character stands for not
05:52 : anything that is not oops anything that
05:56 : is not a single double quote that's this
06:01 : is a double quote and it needs to be
06:03 : escaped this that escape is for nearly
06:06 : itself and then the second one is escape
06:10 : of the slash itself so this guy is a
06:13 : single slash and this guy is a single
06:16 : double quote so so this character class
06:18 : gives us the same thing that gives us
06:20 : basically any character that is not the
06:22 : double quote or the slash
06:25 : so that's what a character is or a
06:27 : character could be a single slash again
06:30 : the single slash needs to be escaped for
06:33 : the sake of nearly and then something
06:36 : called the escape which will be defined
06:38 : thus and we'll define escape as either a
06:45 : single double quote again it needs to be
06:48 : escaped in which case we will return a
06:53 : JavaScript representation of a double
06:56 : quote we're going to sort of translate
06:58 : on the fly here and this is a little one
07:04 : literal slash which needs to be escaped
07:07 : by itself and then in JavaScript we
07:10 : return the same thing one literal slash
07:14 : which also needs to be escaped by itself
07:17 : in JavaScript as well and then we got a
07:23 : forward slash so you could potentially
07:27 : escape the fort slash with a back slash
07:29 : but that's actually optional so you can
07:33 : write it like this as well and then we
07:35 : got a bunch of whitespace character so
07:38 : slash B is backspace and then we got
07:42 : slash F is what is that
07:45 : I forgot what F is but n obviously is
07:49 : new line and R is the character returned
07:52 : and he is the tab there's the U which
07:55 : allows you to specify a character code
07:59 : in hexadecimal numbers but let's do that
08:02 : a little bit later because that gets
08:04 : more complicated we have to deal with
08:06 : hexadecimal numbers etc okay now now
08:12 : we're gonna try to test this out so can
08:14 : we make a have a quote so this is a
08:20 : string which starts with a double quote
08:23 : and ends with a double quote with the
08:25 : content of the string being a single
08:28 : double quote which is escaped by a slash
08:32 : and the reason this needs to be a double
08:34 : slash instead of single is because we
08:36 : want
08:38 : javascript because this is a javascript
08:40 : file we need this javascript file to
08:43 : interpret this guy as a literal single
08:46 : slash so we need to escape it with
08:48 : itself this is gonna get complicated so
08:52 : we're gonna actually externalize this
08:54 : input in a bit but now let's see what
08:57 : happens here okay well have the terminal
09:02 : I'll switch to the replay terminal to
09:04 : see what white kid did here okay so now
09:09 : he's gonna run this parser after having
09:11 : already regenerated the parser that
09:13 : doesn't look right and I think he has a
09:15 : bug in there because he's forgot to
09:18 : transform the data that's parsed by this
09:23 : rule and to just provide a transformer
09:28 : so he's gonna do that
09:30 : so this transform basically takes this
09:33 : array of him information yeah so again
09:37 : think of this as an array and takes this
09:40 : is the 0th one this is the first one it
09:43 : takes the first one so whatever escape
09:45 : is which would have come from one one of
09:49 : these guys one of these transformers
09:51 : would have returned a character back and
09:55 : that's gonna go into data index 1 yeah
10:00 : so basically whatever that Escape
10:02 : returned this character will be that so
10:08 : we're gonna need to keep forgetting and
10:12 : typing in there so run the generator
10:15 : generate parser again and run it and now
10:17 : we get a hello that has literal double
10:21 : quotes in it and that's what we want
10:23 : now let's externalize this JSON code so
10:27 : that we can write just one slash instead
10:30 : of having to write two slashes in there
10:33 : so externalize and now in the persuade
10:36 : us we're gonna have to read in the file
10:41 : which for that we like to use a library
10:43 : called MZ highly recommend this for an
10:49 : and him you're writing nodejs programs
10:52 : really and having to deal with
10:53 : asynchronous stuff so yeah so you
10:58 : required the file system from MZ instead
11:01 : of the normal node.js file system and
11:04 : now you can use async/await syntax which
11:10 : I love by the way we also made some
11:13 : videos about a single weight so you're
11:17 : welcome to check those out as well so
11:19 : this is how you read in a file the
11:22 : result because of MZ the result is a
11:24 : promise with which we can await on the
11:27 : result of which is a buffer object which
11:29 : we can do a to string in order to get
11:31 : the string content and now this is the
11:33 : input we're gonna feed that input into
11:35 : the parser and then parse again and now
11:39 : we get the same result only now the
11:42 : input comes from this input that JSON
11:44 : file let's change the input a little bit
11:49 : put a newline character in there using
11:52 : this slash pen also let's just display
11:59 : the result pick the result out of the
12:02 : parse or that's a type whole yeah okay
12:05 : and then yeah and then we see that now
12:08 : running the parser gets us the string
12:11 : with a valid newline character because
12:14 : we see the lion separation out between
12:16 : hello and the world you can also see
12:18 : that escaping of the double quote
12:21 : character now works
12:22 : cool obviously we should be able to do a
12:27 : race still and we can see that we can do
12:30 : that as well okay and you can do taps as
12:37 : well so that proves we can do the
12:40 : escapes now let's go back and do
12:43 : hexadecimal numbers which on the JSON or
12:48 : JSON data web page it is written in this
12:53 : way so if you're gonna to escape so this
12:56 : actually looks like this
12:59 : it looks like if you have a string and
13:04 : you try to escape slash and then au and
13:08 : then some number say zero zero six five
13:12 : actually that's not a great example yeah
13:15 : there's these are digit these are hex
13:19 : digits and they're gonna be four of them
13:21 : so one two three four each one is a hex
13:24 : digit what's a hex digit or a hex digit
13:26 : can be a digit a normal digit decimal
13:30 : digit or because it's a hexadecimal
13:32 : number it can also be a letter between a
13:35 : and F so yeah and these four digits will
13:40 : give you a character code which you can
13:43 : use for example you can do this and you
13:51 : get one character that looks like a wire
13:53 : with two dots above it I don't know what
13:56 : that means exactly but if you do
13:57 : something at a lower range like that
14:00 : actually got a C for the hexadecimal 0 0
14:05 : 4 3 so that is what the JavaScript
14:08 : language supports and also the JSON
14:11 : language supports so we've got to
14:12 : support that okay so let's write those
14:15 : rules in so yeah we're gonna have a if
14:20 : you get gonna have a escape slash and
14:22 : then I you the parser is gonna expect
14:25 : four hexadecimal digits what's a
14:27 : hexadecimal digit it's either a digit
14:30 : the normal digit
14:31 : decimal digit or one of the letter
14:35 : between ans and for a digit let's just
14:41 : return well but we're gonna actually
14:43 : convert it to a number and then over
14:45 : here we do a calculation to convert it
14:48 : to decimal number here but this is how
14:51 : it works
14:52 : so we're gonna say we really gonna take
14:54 : whether it's lowercase or uppercase
14:56 : we're gonna lowercase it first to put it
14:59 : in our switch statement and we're gonna
15:00 : say if it's an a but that's gonna be the
15:04 : 10 if the B is 11 and so forth all the
15:07 : way up to F with
15:09 : fifteen so now Oh actually there's a
15:13 : problem here the digit
15:15 : I think he's gonna fix it and just a
15:17 : minute there so if it's a decimal digit
15:20 : we're just gonna use the number
15:22 : conversion function to take this digit
15:26 : and convert it to a number without
15:28 : having to read this switch statement now
15:31 : we need to convert this so like let's
15:34 : say there's a there's four digits
15:37 : basically at this point
15:41 : [Music]
15:43 : there's four digits but this one
15:47 : represents one but this one actually
15:50 : represents 16 this one actually
15:54 : represents 16 times 16 256 oh you can
16:03 : tell I'm not a real computer scientist I
16:07 : can't just do that in my head and that
16:11 : one actually stands for 16 times 16
16:13 : times 16 which is 4096 so so busy in
16:25 : order to convert this I want to multiply
16:27 : this guy by 16 and then half this guy
16:32 : and then put them together and multiply
16:33 : that by 16 and then put that together
16:37 : and then multiplied that by 16 and then
16:41 : finally put add it to that guy I hope
16:46 : that made sense now let's do it in the
16:50 : code so so if you look at this match has
16:56 : an array like that then does U is index
17:01 : 0 and then this is 1 2 3 4 right so so
17:07 : we're gonna take this digit here by the
17:10 : 1 and put it in here multiply it by 16
17:13 : and then Plus this second one
17:17 : and then add them together and then
17:20 : multiply the result of that all of that
17:22 : by 16 now I incorporate the third digit
17:26 : add them together and then add them all
17:30 : together again and multiply by 16 and
17:32 : then finally at the fourth digit and
17:35 : then we have the character code and
17:37 : after we have the character code I'm
17:39 : gonna use this method from the string
17:41 : constructor string from character code
17:43 : and give it the code that should give us
17:45 : a character let's give that a try
17:50 : i generate the parser again there's an
17:54 : probably syntax error code I think it's
17:56 : this one we have an extra extra arrow
18:00 : that should be an or sign okay looks
18:08 : like that was fixed and now we're gonna
18:11 : try to look no that's not right change
18:15 : that to you and now that gives e if we
18:21 : do 0 for 1 in hexadecimal
18:24 : that gives us a capital A let me try it
18:30 : by myself because the replay time is
18:33 : sort of giving me messy help so yeah
18:37 : 0 for 1 gives us an A and if you go to
18:41 : Google for the ASCII table you'll know
18:44 : what to expect for what so this table
18:48 : gives us everything from code 0 to code
18:53 : 127 although this 127 is in decimal
18:56 : numbers what you want to look at is the
18:58 : column in hexadecimal numbers that's X H
19:02 : X here and we can see that with the with
19:10 : the 41 for hexadecimal that gives us a
19:14 : character a so if we wondered and
19:17 : lowercase n and has a hexadecimal 4 that
19:21 : is 6 e let's give that a try
19:28 : yeah the work isn't so very cool he's
19:33 : trying a few different characters let's
19:35 : see what he come up what the heck is
19:37 : that I have no idea what that symbol is
19:39 : that looks really weird I'm gonna
19:43 : conclude this video around here if
19:46 : you're watching if you are a viewer of
19:50 : our channel a whiter than I I like for
19:56 : you to give us some feedback I put a
19:58 : comment under under this video will love
20:02 : to hear from you if you're interested in
20:04 : actually learning how to build a parser
20:07 : using nearly Jes or any other tool if
20:11 : you want to try something different and
20:12 : I strongly recommend that you actually
20:15 : get some hands-on experience and do it
20:18 : on your own if you want to follow along
20:22 : with this video series and build a JSON
20:25 : parser you can do that
20:27 : I cannot intentionally leave left off
20:31 : not building the parser for the object
20:34 : literal syntax so that could be a
20:36 : take-home exercise for you the viewer
20:39 : but you could build other a parser for
20:42 : other languages a lot of their simpler
20:44 : languages - such as CSS or HTML next in
20:48 : this series we're going to start getting
20:51 : into something that requires some
20:54 : computation reduce a parser for simple
20:58 : math equations so thank you for watching
21:02 : and hope to see you in the future

How to Make a Parser with Nearley.js - Part 4 - Parsing String Literals

Transcript