Make a Parser With Nearley.js - Part 3

In this drive by code session we’ll demonstrate how to handle whitespaces in a grammar using Nearley.js. The convention used in the video is based on the recommendations by Kartik Chandra - the creator of Nearley.js. As usual, Waikit drove the code before hand, and I’ll replay his code session and explain what he did.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : hello there welcome to a drive-by code
00:02 : session this is can be a continuation of
00:07 : the series we've been making about
00:11 : building programming languages this is
00:13 : gonna be at the third in the series of
00:19 : introducing you to D parser generator
00:21 : called nearly 2 yes if you haven't seen
00:27 : the first two in which white kit and I
00:31 : show you how to build a JSON parser or a
00:37 : partial json parser then you should
00:41 : check the parts one and two first before
00:45 : continuing on with watching this one so
00:50 : go there now watch them and come back oh
00:53 : wait okay so in this episode we're going
00:58 : to show you how to parse white space or
01:02 : how to deal with white space in your
01:04 : parser let me show you where we're at
01:07 : right now first so at this point we have
01:10 : a parser which was written using this
01:15 : grammar syntax so each one of these
01:19 : production these things is called a
01:22 : production rule and and each of the
01:28 : production rules some some of them are
01:30 : compound production rules like the ones
01:32 : with an or operator this is saying a
01:35 : value can be derived from a number a
01:39 : boolean a minor put of that is a string
01:43 : or array and each one of these rules has
01:47 : a sort of a transformer that transforms
01:51 : the input that was matched to the
01:56 : representation the object representation
01:59 : that you want to use and what you do
02:03 : with this grammar is when you have it
02:06 : you run this little
02:10 : man that we've set up to compile this
02:14 : grammar into a JavaScript file like so
02:18 : and in and then utilize it in this way
02:22 : by writing a little script to wire it up
02:24 : together and now we're able to parse
02:27 : this JSON input here so I'll run this
02:30 : parse js5 and it says par success and
02:34 : the result of parse is this which is a
02:37 : JSON representation not this I
02:40 : represented the JavaScript
02:42 : representation of this JSON input here
02:46 : the thing we're gonna tackle now is
02:49 : adding the ability to handle white
02:52 : spaces which currently does not exist so
02:55 : so what we want basically is to allow
02:59 : the input to have white space like so so
03:03 : that for example we simplify this input
03:06 : a little bit here this is an array still
03:10 : even though it's got some additional
03:13 : white space here and in the middle
03:15 : between the numbers between the elements
03:20 : of the array etc so how do we do this
03:22 : so the the author of nearly his name is
03:27 : Koch he has written up this article
03:30 : called how do grammar good and one of
03:34 : the things he says in the article is
03:36 : that by convention what he likes to do
03:40 : or I guess the community of the nearly
03:44 : Jas users what they like to do is use a
03:46 : single underscore to mean optional
03:49 : whitespace and a double underscore to
03:52 : mean mandatory whitespace and which
03:54 : makes it easy to write rules like this
03:57 : so that's what we're gonna do so let's
04:02 : see again I mentioned that props to why
04:06 : kit
04:07 : why kit is the one that actually writing
04:09 : this code I'm just gonna replay that
04:12 : replay this and explain what he did so
04:15 : now we're gonna define
04:19 : I thought we were gonna define oh let me
04:21 : look at this replay okay so now we're
04:27 : gonna add this rule that this finds this
04:30 : optional whitespace symbol here make it
04:33 : fun slightly bigger okay so this is the
04:37 : symbol that means optional whitespace
04:39 : and we're gonna say is to find us a
04:43 : character class that matches a white
04:46 : space or tabs that is either white
04:51 : spaces or tabs this star as you people
04:56 : who know regular expressions know this
04:59 : guy here that means zero or more
05:03 : occurrences of so so a panel white space
05:07 : is zero or more occurrences of either or
05:10 : white spaces or or taps simple and then
05:15 : we're gonna sprinkle the optional white
05:18 : space wherever we see fit here so the
05:21 : first thing I'll say is in the top level
05:24 : input it's this you can have optional
05:29 : white space around the top-level input
05:31 : around the value that represents the top
05:34 : level input so I put some optional white
05:37 : space here and here one thing to note is
05:40 : that once you once we added this
05:43 : optional white space this this
05:45 : transformer no longer works as expected
05:48 : because of what we have done effectively
05:50 : is take something that used to be simply
05:54 : one one right hand side symbol to
06:00 : something that is now three right hands
06:03 : I symbols and which means when the data
06:08 : is marshaled the transformer will get an
06:11 : array that has three things in it
06:12 : instead of one well as this ID function
06:15 : is expecting one symbol so how do we fix
06:18 : that
06:22 : well we'll have to write longhand for
06:26 : this right arrow function that takes in
06:29 : the data which will receive and again
06:31 : to receive this array here that has
06:36 : three elements in it and I'll grab the
06:38 : middle one because there's zero this is
06:39 : one that's to do one we want his index
06:42 : here okay hmm
06:47 : so that should do it for input another
06:50 : place we want to sprinkle in these wet
06:52 : spaces is at the beginning of the array
06:57 : right after the opening bracket and also
06:59 : right right before the closing bracket
07:04 : that's gonna break this transformer too
07:08 : by the way
07:08 : another another thing we another place
07:11 : we added is in between the brackets for
07:14 : an empty array that's easy and another
07:19 : place we're gonna add it is right before
07:21 : the comma that separates array items and
07:23 : right after it as well so I think that's
07:28 : all of them we don't this JSON parser
07:32 : currently does not support object
07:34 : notation that is meant to be left as a
07:37 : homework assignment for you the viewer
07:42 : okay so so now that we made a change to
07:46 : the grammar we regenerate the parser and
07:50 : let's run this script again oops
07:54 : mistakenly type in the replay terminal
07:57 : again how dare me okay so now with parse
08:03 : success but we're not getting the value
08:07 : we expected this is clearly this
08:09 : supposed to be that an array that has a
08:11 : 1 and a 2 in it we have an array with an
08:14 : empty string in it so clearly white kid
08:18 : has messed that up on obvious thing that
08:23 : was must have been wrong is that the
08:28 : this array has increased in its size by
08:32 : 2
08:33 : due to these additional of these
08:35 : whitespace symbols so that must be a
08:38 : problem so we need to fix this so
08:41 : basically again to visualize this for
08:43 : you
08:45 : this is now an array of five symbols and
08:49 : their index as follows
08:53 : whereas previously before adding these
08:55 : white spaces
08:56 : they this array only has three symbols
08:59 : not so now the thing we want in here is
09:02 : actually index two so white kids gonna
09:08 : fix that there we go okay he regenerates
09:14 : the parser and now it fixed it I think
09:18 : he also needs to fix this one because we
09:20 : added white space here also he didn't
09:27 : fix it though okay I think that's a
09:31 : problem zero one two three four oh he
09:35 : did fix it
09:36 : he changed this to a four okay so he fix
09:39 : this one as well so now with that white
09:42 : space is working we can put them in
09:47 : before the input any number of them I'm
09:52 : gonna I'm gonna try to run this code
09:56 : myself there it still works
09:59 : even though there's white space here in
10:01 : at the beginning of the input and also
10:04 : at the end of the input and if there's
10:05 : white space before the closing bracket
10:09 : or in between the commas oh let me do
10:12 : some taps too so you can have white
10:17 : space in all these places and the parser
10:20 : still works so the parser is able to
10:23 : gracefully handle these white spaces now
10:25 : very cool so that's that's white spaces
10:29 : with nearly and I'm gonna bring this
10:34 : episode to a close next episode will be
10:37 : about correctly parsing strings with
10:44 : with escape code that allows you to have
10:47 : double quotes in your string for example
10:50 : and also things like new lines and
10:54 : characters in the string thank you for
10:57 : watching and I hope to see you next time

Make a Parser With Nearley.js - Part 3 - Handling Whitespace

Transcript