Building MyPL Part 3

In this drive by code session we finish building a programming language called MyPL by writing a generator that takes MyPL AST format and translates it to JavaScript. WaiKit writes the code and I replay it and explain what he did.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : hello welcome to another drive-by code
00:02 : session this is gonna be number five I
00:05 : believe in this series about how to make
00:08 : a programming language or hope to make
00:11 : provement languages in general you can
00:13 : make as many as you want so we are in
00:18 : the process of building a programming
00:20 : language called my pl again white kit
00:23 : wrote the code I'm gonna replay it and
00:25 : explain what he did if you missed the
00:28 : first episodes in the series I recommend
00:31 : that you go back and check them out
00:33 : before continuing with this one so
00:37 : currently where we have built a parser
00:41 : for the programming language and I'll
00:43 : just briefly go over how that works so
00:46 : the programming language is called my
00:48 : Pio this is code in my P my pl I have
00:54 : two examples example one that my PL has
00:58 : two variable assignments example two has
01:04 : a while loop and the print statement in
01:06 : it we have a parser which we can invoke
01:09 : using note parse is the name of the my
01:15 : Pio program and it will parse it and the
01:18 : output of the parse is going to be an
01:20 : ast which again that stands for abstract
01:29 : syntax tree and the format of the
01:33 : abstract syntax tree we representing it
01:36 : as a simple JavaScript objects and in
01:39 : fact it's in JSON format this program
01:43 : the code of this program is represented
01:45 : exactly has this AST so now that we have
01:49 : this parser working we're ready to write
01:55 : another program there we'll take this
01:58 : ast as input and generate JavaScript so
02:02 : that's what this episode is going to be
02:05 : about so this is going to be
02:08 : we're gonna take in code which is the
02:11 : AST it's gonna generate JavaScript and
02:16 : this program that we're gonna build is a
02:20 : transpiler and to flesh everything out
02:27 : so we've done the parser and now we're
02:30 : gonna build the trans power at a yellow
02:33 : box so let's get started so alright so
02:37 : first thing let's make no js' program
02:42 : coordinator yes and I'm gonna kind of
02:47 : guys speed over this code it's gonna use
02:50 : async/await which waked and I also made
02:54 : a series about async wait so you can go
02:57 : check that out
02:58 : if you like but basically what this
03:01 : program is going to do is read in the
03:04 : file in dot ast file which is that's the
03:10 : input file name it's gonna use that to
03:13 : create or generate a dot JS file using
03:18 : the base name of the input file it's
03:21 : going to read in the contents of the ast
03:23 : file there and json dot parse to get to
03:29 : convert it into object form and then and
03:32 : then have this function that takes in
03:35 : the AST form and generates some GS code
03:38 : and then it'll go ahead and help put
03:41 : that GS code it says it wrote a file
03:44 : called example 1j s let me see if that
03:47 : exists it does and it says console
03:51 : dialogue how the world now given a given
03:56 : a program am IPL program can we convert
04:01 : this code into the equivalent JavaScript
04:04 : code let's go ahead and do that well we
04:10 : can input well I mean that input line
04:13 : was a little janky but well it starts
04:16 : small and say let's start with just a
04:18 : variable assignment okay so how do we
04:21 : handle variable assignments let's first
04:23 : loop through these statements because if
04:27 : you look at the ast file let's bring up
04:30 : the ASP ast file for this example this
04:36 : this program just has one variable
04:40 : assignment
04:40 : it looks like that so if we want to just
04:43 : start small and support variable
04:45 : assignments we first need to loop
04:46 : through this array of statements let's
04:51 : see this side by side here so we're
04:56 : gonna use this for loop for off getting
04:59 : fancy using new JavaScript syntax here
05:02 : to loop through these array of
05:05 : statements and I'm gonna say if the
05:09 : statements type is variable assignment
05:13 : we're gonna push a line of JavaScript
05:18 : into this lines variable these lines
05:20 : these are lines of JavaScript and then
05:25 : and they will format the JavaScript code
05:28 : like this will grab the variable name
05:31 : from the ast in this case it's gonna be
05:33 : n here right and then we'll grab the
05:36 : value the initial value also from the
05:38 : ast and then use them to generate this
05:41 : line of job this lets statement in
05:44 : JavaScript at the end gather all the
05:47 : lines and join them together into a
05:51 : JavaScript program so let's see if that
05:53 : works so again the input of the generate
05:57 : program is an AST file and the output is
06:01 : the JavaScript file let's look at the
06:03 : JavaScript file now close this that
06:09 : looks like valid JavaScript to me so now
06:12 : let's have multiple variable statement
06:15 : variable assignment statements and redo
06:18 : the parse and redo the generate and now
06:23 : this JavaScript this multiple variable
06:26 : statements no that's awesome all right
06:31 : now what if the right-hand side of the
06:35 : assignment is a more complex expression
06:38 : like this mm plus n times 3 what's gonna
06:43 : happen now let's try it
06:45 : I think it'll break but so ran through
06:51 : this anyway even though I know
06:53 : gonna work the output looks like that
06:56 : which is clearly not valid JavaScript so
07:00 : let's go ahead and fix that okay what
07:02 : can we do here well let's have a
07:05 : separate function that just generates
07:08 : JavaScript expressions right
07:11 : an expression is what well we know that
07:14 : from the parser from the Grandma an
07:20 : expression expression is something that
07:22 : yields a value right after you evaluate
07:26 : the expression you get in value and the
07:27 : expression can either be an unerring
07:29 : expression which can be a number or
07:31 : identifier or it can be a binary
07:33 : expression which has this operation on
07:36 : it so let's have a function that
07:41 : converts a my Pio statement sorry
07:44 : converts our my Pio expression into a
07:47 : JavaScript expression and if we have
07:49 : that we can just use that convert this
07:52 : expression now this value would be
07:54 : JavaScript expression and we plug it in
07:56 : into this javascript statement here okay
08:01 : so how is this gonna work we'll say if
08:06 : the expression is an object then we're
08:10 : gonna do further stuff with it if it's
08:12 : not an object we're going to assume it's
08:13 : either a string or a number which means
08:15 : it's like a variable reference or just a
08:19 : just a literal number if it is an object
08:22 : we're gonna look at what is the type of
08:25 : the ast and if it's a binary expression
08:30 : I'm going to recursively call ourselves
08:34 : to evaluate the left-hand side and the
08:38 : right-hand side because both the
08:39 : left-hand side and right-hand side could
08:42 : potentially be binary expressions in of
08:45 : themselves
08:46 : so this is where recursion comes in
08:50 : handy and all those years of geeking out
08:53 : on computer science is really paying off
08:55 : now okay so now after you have left hand
08:59 : side and right hand side we're gonna get
09:03 : the operator and put together this line
09:06 : javascript left-hand-side operator and
09:08 : then right-hand side let's see if that
09:10 : works
09:14 : and it does except that the
09:16 : multiplication symbol is not recognized
09:19 : by JavaScript ouch
09:21 : what are we gonna do well let's have an
09:25 : old format that just translates of the
09:29 : operators for us from my PL 2 JavaScript
09:33 : so we'll look up this operator map in
09:37 : order to do the translation and do it
09:41 : again so generate example 1j s again
09:45 : let's look at that and that works now
09:47 : that's valid JavaScript I can tell so
09:51 : very cool we just translated this my PL
09:55 : code into this JavaScript code print
10:00 : statements
10:01 : let's make print statements work so I'm
10:04 : gonna add 3 print statements to this
10:06 : program to see that the output of these
10:09 : this computation here and let's try to
10:15 : generate stuff actually no let's make
10:18 : them work first before even attempting
10:19 : so again using a similar scheme we can
10:23 : say if the statement let's look at the
10:26 : statement let's look at the est so this
10:30 : is the ast now is got some variable
10:32 : assignments it's also got some print
10:35 : statements now so for the print
10:37 : statements we want to be able to handle
10:39 : them so we say if the statement type is
10:42 : print statement we have tool at our
10:45 : disposal this generate js4 expression
10:48 : now we can reuse for this case we can
10:52 : say the expression we're gonna print
10:54 : we're gonna run it through generate js4
10:57 : expression to convert the expression
11:00 : from my pl to javascript so this
11:03 : expression now is an expression in
11:05 : JavaScript and you know how to print in
11:08 : JavaScript console that log
11:12 : easy so now let's generate the
11:16 : JavaScript for this
11:18 : st so input AST output javascript and
11:22 : now that looks actually like valid
11:27 : JavaScript program to me run it and we
11:30 : get answers we have a we have a like
11:33 : living and breathing programming
11:36 : language now let's try example number
11:40 : two which had a while whatever we can
11:44 : get the wild lips to work okay so this
11:46 : is the ast of this while loop right it's
11:49 : got a condition it's got the body which
11:52 : has more statements in it what if we
11:56 : make the generator work with the wild oh
12:00 : how is that gonna go well a while loop
12:04 : is just another type of statement right
12:07 : and the condition again we can piggyback
12:11 : on this generator yes for expression to
12:14 : convert the conditional of the while
12:16 : loop to a JavaScript version of that
12:20 : conditional and as for the body oh my
12:25 : god we can just use this top-level
12:27 : generator is function to generate the
12:31 : JavaScript code for the body because
12:34 : this body right is an array of my Pio
12:39 : statements in the same way that the
12:41 : top-level program is an array of my PL
12:45 : statements so if we had a function that
12:48 : early will take an array of statements
12:50 : and convert it to JavaScript that
12:52 : function will also work for this body so
12:55 : just plug this function right in to
13:00 : reuse again this is recursion - this is
13:03 : Jenna ideas his calling generator yes we
13:06 : just plug it right in to convert the
13:08 : body of the while loop into JavaScript
13:13 : and now we just format this code like so
13:18 : I think he made a mistake forgot to make
13:21 : this body reference dynamic he's gonna
13:23 : fix that here at this point so he's
13:26 : generated this JavaScript code which
13:29 : looks like a valid
13:31 : there's a couple of problems but he's
13:33 : gonna fix that first let's indent this
13:35 : coat which is actually not as hard as it
13:39 : might seem you split up the string into
13:42 : lines is map add some white space at the
13:48 : beginning of each line and then join
13:49 : them back together regenerate it and now
13:53 : we have indentation there's still
13:55 : another problem which is that this end
13:57 : variable has been read Eclair and that's
14:00 : gonna break it everything so we need to
14:03 : not read Eclair at this end and remove
14:07 : this let this is what we want so how can
14:09 : I do that well let's just keep track of
14:12 : all the variables that have been
14:14 : declared so far in in an array and in
14:18 : order for all parts of this generator to
14:21 : to know about this array of variables
14:25 : we'll just pass them in to every single
14:28 : function call there and we'll say
14:32 : initially the array is empty
14:34 : and you can as you look encounter these
14:37 : var statements you'll push into this
14:40 : array so now we can say hey if if this
14:48 : variable has never been declared we're
14:51 : gonna declare it using this let
14:52 : statement and push the variable into
14:56 : this array which keeps track of the
14:59 : declared variables but if it's already
15:02 : in there so else then we'll just have a
15:06 : normal assignment instead of a let
15:09 : statement so let's regenerate the
15:13 : JavaScript and now that looks like a
15:14 : valid JavaScript program but a while
15:17 : loop sort of run that BAM actually works
15:21 : this it seems to start at 2 which is
15:25 : probably not intended so let's fix that
15:28 : by rewriting them my PL code and
15:36 : rear-ending the JavaScript and now it
15:38 : starts at 1 okay that's cool
15:43 : now let's make a run script that kind of
15:47 : allows us to run a my Pio program in one
15:51 : step so at this point we have a run
15:55 : script it's gonna take as input on my
15:59 : Pio file and the output is just gonna be
16:03 : the output of the program because this
16:05 : is gonna run the parser you're on the
16:08 : jet to generate an AST file is gonna run
16:11 : the generator to generate a dot JS file
16:14 : it's also gonna run the note program to
16:16 : execute the JavaScript file all in one
16:20 : step so let's see what happens now let's
16:23 : just skip to the end because no
16:25 : programming language would be complete
16:28 : if you can write the Fibonacci series in
16:31 : it so it's write the Fibonacci series
16:33 : now so alright maybe not as serious
16:40 : we're gonna loop from one to ten
16:42 : basically this is this story the eye is
16:45 : the loop counter and n is where to stop
16:50 : so we're going to keep track of the
16:53 : current number in the Fibonacci series
16:56 : and the next number after it usually you
16:59 : call it f of I and F of I plus 1 I'm
17:03 : gonna call them fi and FII and then
17:07 : let's have this while loop that's like
17:11 : gonna go up from i from 1 to n which at
17:15 : the present is 10 I'm gonna print out
17:18 : the current number in this series which
17:20 : will start at 1 and then when we do this
17:24 : swap thing need this temporary variable
17:26 : to perform this swap now so this is I
17:29 : got at and swap operation so now F of I
17:34 : goes into it if I I goes into F of I and
17:37 : F of I gets the value of adding the two
17:40 : numbers together run that we have an
17:44 : infinite loop because white if forgot to
17:47 : increment
17:48 : here inside the loop so he's going to
17:51 : fix that really quick the reason we got
17:54 : this gnarly error is because the
17:56 : infinite loop generated so much output
17:58 : that the nodejs is child process command
18:04 : ran out of buffer space so we've run
18:09 : that again and now we have the fibonacci
18:12 : sequence let's bump that number a little
18:14 : bit and we can get even more numbers so
18:17 : there is proof that we have actual
18:20 : working programming language I hope that
18:24 : you enjoyed watching the creation of
18:27 : this programming language which we
18:29 : called my pl man whitetip we're gonna
18:33 : continue making this series and we'll
18:35 : dig into more details about how parsers
18:38 : are made and how code generators are
18:40 : made and in general the art and craft of
18:44 : making programming languages thank you
18:46 : for watching

Building MyPL Part 3 - Generator

Transcript