Make Your Own Language 9: Comments and More Ambiguous grammar

This is the ninth episode of a series where you build your first programming language, using these videos as a guide. We’ve setup the syntax for a small programming language. You can clone this project and start getting to work. The project repository is https://github.com/airportyh/smallang. In this episode, add comments and start on working a code challenge by get side-tracked on debugging some more ambiguous grammar.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : hello there this is toby welcome back to
00:02 : the series
00:03 : where we make a programming language
00:06 : together
00:07 : uh we're making a programming language
00:08 : called smalling
00:10 : which is a as the name conveys
00:14 : very small programming language that you
00:16 : can build yourself
00:17 : as your first programming language we
00:19 : have built out
00:20 : most of the language this time we're
00:23 : going to
00:24 : implement comments so that you can
00:27 : comment your code
00:29 : somehow i felt the language wouldn't be
00:33 : complete without the ability to have
00:35 : comments in your code
00:36 : and then hopefully we get done early
00:39 : with that
00:40 : and then we can actually solve a
00:43 : code challenge or two which will most
00:46 : likely
00:47 : result in finding goals in the
00:50 : programming language and have to
00:51 : add features to it and we'll
00:54 : tackle those as we go okay so first
00:58 : thing
00:58 : let's implement comments in the language
01:01 : the lexer
01:02 : already has a rule for comment we just
01:05 : stole that right out of mood.js and now
01:08 : we're gonna
01:09 : utilize that in our grammar let's open
01:11 : up the grammar file
01:13 : and let's say we're going to allow
01:17 : line comments in the same level as our
01:20 : statement
01:21 : so perhaps a comment is allowed
01:24 : here like so let's just see
01:28 : what that gives us i i would expect that
01:31 : something like this would be allowed so
01:33 : let me make a new file
01:35 : example six i'm just gonna
01:38 : copy of the last program except i'll add
01:41 : a couple of comments
01:44 : okay so let's see if we can parse this
01:46 : file now
01:48 : so we generated the parser now let's
01:50 : parse the example
01:52 : number six and it was happy with that so
01:55 : let's look at the ast
01:57 : and there it is a comment so great
02:00 : uh if we generate the code now
02:04 : uh it says unhandled ast note comment
02:07 : well we could spit out the comment in
02:10 : our javascript code i'm going to
02:12 : actually choose to ignore it let's put
02:15 : in
02:16 : a clause for handling comments
02:19 : but i'm going to just ignore it by
02:21 : returning an empty string
02:23 : let's see if it likes that let me see
02:26 : what
02:26 : came of that that looks just fine
02:29 : okay now that we are able to write
02:31 : comments in our code
02:33 : let's solve a code challenge i'm gonna
02:35 : do this
02:36 : roman numerals encoder very
02:40 : classic kata one should map to
02:43 : i and five should map to v
02:46 : so forth i haven't done this in a while
02:49 : so i might
02:50 : have to uh figure this out from first
02:53 : principles all over again oh i just
02:56 : realized that our runtime doesn't have
02:59 : any string
03:01 : functions we're definitely gonna need
03:03 : some string
03:04 : functions i think let's add a concat
03:07 : function for strings although i suppose
03:10 : we could just use the add
03:11 : but i actually prefer having a concat
03:14 : function for strings
03:15 : i'll say s1 to
03:19 : to be explicit that this is for strings
03:22 : i'll call this roman that small i guess
03:25 : i'll call this function to roman it
03:28 : should take
03:28 : a number n this is actually kind of a
03:32 : complex problem
03:34 : let's just start simply say if number is
03:36 : equal to
03:38 : one then we want to return the string
03:42 : i else if number is
03:46 : 2. wow this is gonna get a little
03:49 : tedious i think
03:59 : okay let's start with that
04:04 : okay uh let's print
04:08 : one goes to two roman
04:12 : one let's do this up to five
04:17 : okay
04:21 : uh we have an ambiguous grammar detected
04:25 : ouch okay let's compare the two
04:32 : there's actually no difference between
04:34 : two so
04:35 : that means again it has something to do
04:37 : with white space
04:39 : i'm thinking the ambiguity might be
04:42 : having to do with
04:43 : this body because i did test our
04:45 : fibonacci
04:46 : file which did not have this bracket
04:50 : for the body so i'm actually going to
04:51 : check the grammar
04:53 : for the bracket version which is that
04:56 : one
04:56 : and this one says this needs to be a
04:59 : mandatory
05:00 : line break here and then we use a
05:03 : statement
05:04 : over here oh statements can have
05:08 : white space at the beginning and the end
05:11 : i see
05:11 : so this is white space here
05:15 : and then we have statements and then we
05:18 : have
05:18 : white space so this way space is a
05:21 : mandatory line break which means there's
05:24 : like one or more new lines
05:26 : and both here and here
05:29 : plus some option always based on either
05:32 : side more or less
05:33 : there can be more than one new line
05:35 : actually but in essence
05:37 : this is a white space and this is a
05:39 : white space but then
05:41 : within statements which is defined here
05:43 : they can also be
05:45 : multi-line white space at the beginning
05:46 : and the end so the parser is confused
05:49 : because
05:50 : when it sees white space it could either
05:52 : match it as this
05:53 : guy here or you could also match it as
05:55 : this guy
05:56 : because both of those things are uh
05:59 : there's some
06:00 : optional element to it so i think what
06:03 : i'm gonna do
06:04 : is remove this white space from both
06:07 : sides of
06:07 : this rule here but have a new rule
06:11 : i'll call a program which is the top
06:13 : level
06:15 : that would have this white space
06:18 : like that the first rule of a nearly
06:21 : grammar file is always the top level
06:24 : um and then i'm going to remove it from
06:26 : here
06:27 : now i need to be careful to shift my
06:30 : indexes
06:31 : so this one will be one now and this one
06:33 : would be zero now
06:35 : but this one stays the same because the
06:38 : it's the index within this
06:39 : nested array here and then this
06:43 : lambda body would still reference this
06:45 : statements which
06:46 : now no longer has the leading white
06:48 : space or the ending white space so i
06:51 : think this should
06:52 : solve the ambiguity let me try it
06:56 : yes that solved the ambiguous grammar
07:00 : problem however we now have a unhandled
07:04 : note type undefined so let's see what
07:07 : that is
07:10 : and we don't even get a type
07:19 : let's print the entire node or j sonify
07:23 : it
07:24 : let's just print out the whole thing and
07:26 : see what it is
07:28 : okay it turned out to be an array
07:31 : so there's something of type new line
07:35 : that uh made its way into the ast should
07:38 : not be too
07:39 : hard to diagnose this because it should
07:41 : be in an ast
07:42 : and there it is so i'm gonna search just
07:45 : in case there's multiple ones of these
07:47 : and there are there are multiple ones of
07:50 : these new line things
07:51 : snuck into the ast oh
07:54 : because we haven't written a converter
07:56 : for our program
07:59 : i'm gonna return index one
08:02 : so i think that should fix it
08:06 : and it did okay great uh let's
08:09 : run an example number six
08:12 : great now let's run the roman program
08:18 : okay the roman.ast still has this
08:22 : problem we have some new line characters
08:24 : oh this time is in the body of a
08:28 : lambda i see so we still have some bugs
08:31 : in this code it's definitely this case
08:33 : where we have
08:34 : the brackets surrounding the lambda body
08:38 : and that's not as well tested so oh
08:41 : it's the index problem again this is an
08:44 : array
08:47 : so this is index zero this is one this
08:50 : is two not three
08:51 : but i have three here this should be two
08:55 : that's actually a quite a common mistake
08:59 : that i tend to make okay all right
09:02 : at the beginnings of a roman numeral
09:05 : converter program
09:06 : it can do one through five so far
09:09 : suppose what we could do is if it's more
09:12 : than
09:13 : five let's then we just subtract five
09:16 : from it if it's six through nine
09:18 : subtract five
09:20 : from it call ourselves put a v
09:24 : and then concat v with the result of
09:27 : calling to
09:28 : roman with this yeah i think that
09:31 : probably works this would be like al's
09:35 : cat v with two roman
09:39 : number minus five the parser isn't happy
09:43 : now
09:45 : oh obviously we can't use the subtract
09:48 : operator we have to say
09:50 : subtract
09:53 : uh okay that doesn't
09:57 : totally work nines is actually supposed
10:00 : to be ix
10:01 : this is actually kind of a challenging
10:04 : problem i
10:05 : am gonna actually take a break here and
10:07 : then perhaps continue
10:08 : next morning