LLVM Tutorial #22: Custom Operators

In this series I walkthrough the LLVM "Kaleidoscope" Tutorial, where you follow step by step to create your first programming language frontend using LLVM as the backend. Last time we finished implementing for loops in chapter 5. This time we start 6 chapter where we'll implement custom operators in the language.

Transcript

The following transcript was automatically generated by an algorithm.

00:01 : hello there welcome back to the llvm
00:03 : tutorial
00:04 : walkthrough where we're going to uh
00:08 : we have been work walking through the
00:10 : llvm
00:12 : my first language front end with llvm
00:16 : tutorial
00:18 : we have made it past the first five
00:20 : chapters
00:21 : and uh so right now we're gonna
00:24 : try to go after chapter six
00:28 : so let's do it um
00:32 : okay so chapter six is about this
00:35 : idea of user-defined operators
00:38 : as in you can
00:40 : [Music]
00:43 : you can custom define what the plus
00:46 : means
00:46 : i believe and maybe even other operators
00:49 : like
00:50 : you can define the add symbol
00:53 : to be a custom operator
00:56 : uh you can have honorary operators or
00:59 : binary operators
01:00 : so these guys are binary operators
01:02 : because
01:04 : a binary operator takes two operands
01:08 : a plus operator has a left hand side and
01:10 : a right hand side and that's two
01:12 : operands
01:13 : so these these guys are binary
01:18 : uh unreal operators have one operand so
01:22 : the exclamation point
01:26 : like exclamation point like you you
01:30 : do something like if it not completed
01:33 : so that guy is an array operator
01:37 : okay so so the
01:40 : the scheme that we're gonna take um
01:43 : following
01:44 : chapter six is that we're going to be
01:46 : able to
01:48 : custom define either unary operators or
01:51 : binary operators and you would write a
01:55 : unreal operator like this
01:58 : uh where you you first
02:02 : write either the word only or binary
02:06 : to to to name the type
02:09 : and then you write the symbol
02:12 : of the operator in this case the
02:14 : exclamation point
02:16 : and here there's the greater than um
02:22 : and and then you write some code which
02:25 : is going to execute when that operator
02:28 : is used uh so that's
02:32 : the idea we're going for this syntax
02:35 : we have this def i believe we had a def
02:40 : for defining functions
02:44 : is that true yes we already have the dev
02:46 : tokens
02:48 : we have to already have the dev token
02:49 : for defining functions in the style of
02:51 : python
02:52 : but now we're going to use def to define
02:55 : custom operators
02:57 : as well that's the idea so um
03:01 : let's write the tokenizer for it so
03:05 : um operators what are we gonna do so
03:09 : we're gonna have oh we're gonna add two
03:11 : tokens
03:12 : the the these keyword the unreal and
03:15 : binary are
03:16 : keyword tokens so we're gonna add those
03:18 : in
03:19 : into our tokenizer so
03:23 : that makes sense
03:31 : okay and then we're gonna add these two
03:35 : clauses
03:36 : to um to this if statement
03:41 : down below where
03:45 : where when when whenever we parse an
03:48 : identifier
03:50 : we exclude the keywords from
03:54 : from being an identifier so this is
03:57 : grabbing alpha numeric
04:01 : identifiers and after it grabs an
04:04 : identifier
04:05 : we have to check whether the name
04:08 : is one of our keywords and if it is one
04:10 : of our keywords
04:12 : we can't allow that to be an identifier
04:15 : so this is
04:16 : this is the exclusion uh
04:20 : logic here so i'm just gonna copy this
04:26 : in
04:32 : i'm gonna put in
04:37 : the blocks okay so that will give us the
04:40 : binary and under a uh
04:42 : keywords as seen here and here
04:47 : and then we
04:50 : are what is this thing doing here
04:53 : uh oh we're gonna need to update the
04:56 : prototype
04:58 : ast node uh let me just remind you
05:01 : what the prototype ast node is it's the
05:05 : prototype of a function so
05:09 : previously to define a function you
05:12 : would
05:13 : write something like this
05:18 : the def my fun
05:22 : i believe uh and then you have a
05:25 : parenthesis
05:26 : list and then or a parameter list i
05:29 : should say
05:31 : uh so this guy is the prototype
05:35 : of a function so when we're dealing with
05:37 : uh
05:39 : when we're saying prototype ast it's the
05:42 : ast
05:43 : node that represents this part of the
05:46 : statement
05:46 : not including the body of the statement
05:50 : um so now we're gonna basically say
05:54 : well it's more complicated than that
05:56 : because we can have these custom
05:58 : operators in this prototype syntax
06:02 : we can have this under a thing and this
06:04 : binary thing and this
06:06 : this uh
06:10 : symbol and there's even this number here
06:14 : uh if when in the case of the binary
06:17 : operator
06:18 : there's this number which gives the
06:20 : presidents of that
06:22 : operator or allows the user to define
06:26 : the presence of the
06:27 : operator and we
06:31 : currently have handling for
06:34 : presidents of operators and
06:37 : they're defined as such
06:40 : um
06:43 : like uh multiply and divide have higher
06:46 : presence
06:46 : than plus and minus and then which in
06:49 : turn has
06:50 : higher precedence than less than and
06:52 : greater than
06:53 : i actually believe the tutorial has only
06:55 : defined one
06:56 : of less than and greater than i didn't
07:00 : define
07:00 : greater than i believe but whatever
07:05 : i i think is probably left one of them
07:07 : out so that it can be implemented as a
07:10 : custom
07:10 : operator is what i'm guessing but anyway
07:14 : so let's add this new code to this
07:17 : prototype ast
07:24 : [Music]
07:28 : um this is the code for parsing it
07:33 : but i think we're gonna change the class
07:36 : i believe
07:39 : where is the class there it is okay
07:42 : so it used to be like this and we're
07:45 : gonna actually add a whole bunch of
07:47 : stuff
07:48 : to it uh including
07:51 : a boolean a flag that says it is an
07:54 : operator or not
07:57 : unsigned president in the case that it
08:00 : is
08:01 : a binary operator it it's going to have
08:04 : a
08:04 : precedence which is an unsigned number
08:08 : um and then we're gonna need to
08:12 : add those things to the constructor
08:19 : so let's do that so there's a bull
08:22 : is operator
08:26 : and unsigned
08:38 : precedence
08:44 : and then this is the initializer list
08:47 : so i'm going to write this in
08:49 : initializer list
08:52 : syntax
08:55 : okay um and then
08:58 : looks like we're gonna
09:02 : make a couple of methods to tell you
09:04 : whether it's a honorary or a binary
09:06 : operator that's fine
09:10 : we can do that
09:17 : i have not seen this const
09:21 : syntax before i don't know if that means
09:25 : it's a constant
09:28 : method or what
09:32 : this looks really weird
09:37 : if you know about the syntax please
09:40 : enlighten me i don't know what this
09:43 : const is doing here
09:50 : okay so if the arcs.size is
09:53 : two then it's a binary operator
09:56 : otherwise it
09:57 : if it's arc size is one it's a
09:59 : unreoperator
10:01 : and then we're gonna have a get
10:06 : operator name method which is also
10:09 : const
10:15 : maybe it means uh i think it means
10:19 : this is doesn't mutate the
10:22 : variables in the class
10:25 : that's probably what this cons means
10:28 : like by
10:28 : calling is unreal up you're
10:32 : not modifying anything
10:35 : at all you're just reading
10:46 : okay so if they if they're just gonna
10:49 : err out if
10:50 : somebody tried to make an operator that
10:53 : has
10:53 : three arguments
11:01 : okay this name
11:05 : is referring to this name
11:12 : uh this name okay
11:15 : uh that's just getting the
11:18 : last character
11:22 : of the operator of the name
11:25 : the last character and it assumes that
11:27 : an operator
11:28 : has one character i believe
11:34 : hmm that's suspicious like what what if
11:37 : we wanted
11:39 : an operators that have
11:42 : more than one um
11:46 : character in it i wonder
11:50 : what we would have to change to get that
11:52 : working that might be a bonus
11:54 : extra credit thing that we might attempt
11:58 : [Music]
12:08 : later
12:14 : okay
12:17 : so this is just return so we basically
12:19 : have these
12:20 : assessor methods to ask
12:24 : whether this what kind of operator is it
12:27 : all the information about the operator
12:29 : that would need
12:31 : and then we're gonna need to parse it so
12:34 : let's go to
12:35 : parse prototype
12:38 : and we're going to update this this is
12:40 : going to get a lot more messy
12:42 : now this part prototype function
12:46 : will get quite messy so this eats a
12:51 : token um wait
12:59 : i think it's already eaten the death
13:02 : token at this point so i mean it looks
13:05 : something like def my fun
13:09 : x y x y right it looks something like
13:12 : this
13:13 : or it could look like binary
13:16 : exclamation point five
13:20 : x y let's do
13:26 : star let's do add sign
13:31 : okay or it could be oneri
13:37 : which underneath operators do not have
13:40 : the five
13:40 : so i'm gonna maybe uh maybe the pound
13:43 : sign this time and this
13:44 : has to have one uh
13:47 : but i i think this the the part that
13:50 : this
13:51 : method parses excludes the deaf
13:54 : uh token because i don't see the word
13:58 : deaf
14:01 : um referenced here
14:04 : yes i was right about that because the
14:07 : preceding line
14:08 : to calling parts prototype already ate
14:11 : the death token
14:13 : okay so i'm gonna get rid of the
14:16 : def here so these are the examples
14:20 : of what this method is going to parse
14:24 : um and we need some conditionals because
14:28 : it could either be
14:29 : that or that or that right so
14:32 : okay
14:43 : all right let's go through this so
14:46 : um i have a kind
14:51 : function uh
14:55 : which is zero
14:58 : we're going to figure out the kind
15:02 : based on what type of thing it is
15:07 : okay that's fine
15:10 : and then it's going gonna set the kind
15:12 : if depending on the token type
15:15 : we're gonna set the kind of thing we
15:17 : have
15:18 : and then later on we might need to ask
15:21 : it what kind it is
15:24 : i think that's fine okay
15:28 : i'm going to follow suit there's a
15:31 : function name
15:34 : there's this function name here oh there
15:36 : there it is
15:37 : okay all right let's do it like this
15:43 : unsigned kind
15:46 : zero and i'll just copy this
15:50 : comment for documentation sake
15:56 : and then the binary precedence
15:59 : in the case
16:02 : that uh it's a binary operator we're
16:05 : gonna have a president's number we're
16:06 : defaulting it at
16:08 : 30 and then
16:11 : we're not gonna eat the token just yet
16:15 : i don't believe we're gonna switch
16:18 : on the cursor token to see what type it
16:21 : is
16:22 : i usually prefer to have the default
16:24 : statement at the end
16:27 : so although i
16:31 : you know i'm just gonna do it
16:34 : my way
16:41 : so if the token is an identifier
16:44 : we know it's not a binary or
16:47 : we know it's not the only or binary
16:50 : keyword because if
16:51 : it were unreal or binary we have special
16:54 : token types
16:55 : for that
17:01 : and then we're going to advance the
17:03 : token and break out of this
17:05 : switch statement if it's a binary token
17:14 : we're going to advance the token in
17:19 : okay we're expecting an
17:23 : is ascii okay is ascii i think that
17:30 : means
17:36 : it's a valid character oh because
17:38 : negative numbers
17:39 : are not ascii so that this is sort of
17:42 : like testing
17:44 : for the case where it's negative because
17:47 : it might be negative
17:48 : if um
17:52 : it might be negative if it were any of
17:55 : these tokens because we made them
17:57 : negative that was a little trick
18:00 : that the author of the tutorial is using
18:04 : if it's negative means it's one of our
18:06 : custom tokens
18:08 : types but if it's non-negative
18:11 : or is ascii then it
18:14 : actually represents a
18:18 : actual character
18:30 : okay and then
18:47 : the function name is going to be binary
18:52 : oh and then plus the that character
18:56 : whatever that
18:57 : is
19:05 : okay and then we're gonna read the
19:08 : president's
19:09 : number if it's present
19:21 : so this advance the token so we were
19:24 : looking at this
19:25 : token here well here we were looking at
19:28 : this
19:29 : token and then
19:33 : did it right get next binary this should
19:36 : be get next token
19:38 : and then we advance this token and then
19:42 : now we're looking at this
19:43 : operator token uh and then now we
19:46 : advance the token again
19:48 : and now we're looking at this number
19:50 : token which
19:52 : we're gonna use that to represent the
19:54 : presidents of the
19:56 : binary operator
20:06 : okay so if the president's number is
20:10 : not within the valid range we're gonna
20:13 : error out
20:17 : um otherwise binary presidents
20:22 : is unsigned
20:26 : we're going to cast them out to unsigned
20:30 : the num val i guess
20:33 : is put there by the lexer actually this
20:36 : is a global variable
20:43 : okay and then we're going to advance the
20:45 : token and then
20:48 : break out of here
20:51 : um let me think
20:56 : what about the honorary type
21:05 : and then there's a default case
21:08 : don't forget about that
21:22 : return no pointer i prefer returning the
21:24 : snow pointer
21:26 : explicitly uh why did it not
21:29 : test for the honoree
21:42 : type
21:44 : huh uh
21:47 : is that that might be a mistake i don't
21:50 : see any reference to the honorary
21:52 : operator type okay
21:56 : if that's the mistake then i could check
22:00 : at the end of the code listing at the
22:03 : end of this
22:04 : page
22:09 : i'm gonna search for
22:14 : pars prototype
22:20 : yeah yeah yeah they they they're
22:23 : supposed to have this
22:25 : toke honorary clause here
22:28 : they had this in the full code listing
22:34 : but not in the middle of the article
22:38 : whoops okay i'm gonna go ahead and
22:42 : copy that
22:56 : excuse me
23:32 : okay this similar thing to the binary
23:35 : operator
23:36 : except that we didn't have to deal with
23:37 : this president's
23:39 : thing okay so
23:42 : after that what we're going to do
23:48 : is um let's see i'm gonna eat the
23:51 : identifier
23:53 : i don't think we're gonna do that here
23:56 : anymore
23:56 : because we did it inside this
24:02 : um switch statement
24:08 : and then okay we're here matching this
24:12 : place in the progression of things
24:16 : and then this is gonna read in the
24:19 : argument names i don't think that part
24:22 : changes
24:22 : at all
24:26 : but finally we have this if the kind
24:30 : make sure that there's a right number of
24:32 : names
24:34 : for the operator so
24:38 : so i think this is saying if kind
24:42 : which that means if kind is not zero
24:45 : i'm going to be more explicit and right
24:47 : if kind is not zero
24:50 : which means if it's zero it's an
24:53 : identifier it's a normal function but if
24:56 : it's one
24:57 : or two then it's an operator
25:03 : and our names dot size
25:07 : is not equal to kind
25:10 : that's clever the uh the the number
25:13 : the kind number happens to equal to the
25:16 : number of
25:17 : arguments uh of that the
25:20 : operator would take so if that they're
25:23 : not equal
25:25 : then we're going to error out
25:39 : okay i think we have to pass
25:43 : in the things
25:48 : including the
25:52 : uh whether it is a operator or not
25:57 : uh they just wrote yeah kind
26:00 : is not zero that's a boolean and the
26:03 : binary
26:05 : presence if that exists
26:08 : okay now we're gonna plug this
26:11 : in wait
26:14 : oh okay we're gonna go back here
26:20 : and follow the main flow of the tutorial
26:22 : now whoa
26:25 : we have a mandelbrot set going on that's
26:28 : amazing
26:29 : okay can't can't wait until we get to
26:31 : that point
26:32 : uh okay um
26:36 : so we just did this but
26:39 : we we followed the code in the main
26:42 : index
26:44 : now we're going to write the code
26:47 : generator
26:51 : um maybe we break it up actually because
26:54 : that's what i've been doing in this
26:56 : series
26:57 : so i'm gonna stop this video here and
27:00 : then the next one will
27:01 : implement the code generation see you
27:05 : there