How GUIs Work 1: Graphics and Events

I have recently implemented my own GUI framework/toolkit for terminal-based programs and learned a great deal about how GUIs work. This is the first in a series of videos in which I attempt to explain how GUIs and UI frameworks work underneath.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : we want to talk about how
00:02 : a ui framework works
00:05 : using my ui framework as an example but
00:09 : i want to talk about it in general as
00:11 : well so sort of like how we did the how
00:14 : databases work
00:15 : we'll try to explain how gui's work okay
00:19 : first thing
00:20 : i want to describe is actually as as a
00:23 : implementer of a gui what layer are you
00:27 : using to make your gui work i think
00:30 : there's two separate things one one is
00:32 : the graphics layer
00:35 : so i'll call this the lower layer
00:37 : as in the the gui is a higher layer it
00:41 : is a definitely an abstraction as we
00:44 : talked about in the
00:46 : leaky abstractions episodes
00:49 : a gui is a very big abstraction actually
00:52 : it's trying to give you this
00:54 : illusion that you have these
00:56 : ui components
00:59 : that you can
01:00 : combine and do different things when
01:03 : there is an abstraction is trying to
01:05 : hide a lower layer so what are the lower
01:07 : layer
01:08 : there's a graphics layer
01:11 : and there's the uh
01:13 : user events
01:15 : keyboard
01:17 : mouse
01:18 : others so i would call these two things
01:22 : the the two lower layers
01:25 : that i'm programming against and when
01:27 : i'm saying the graphics layer you can
01:29 : think of something like
01:31 : the html5 canvas api
01:35 : where where
01:36 : where you're allowed to draw things you
01:39 : can say draw
01:41 : a line
01:42 : you can say draw a rectangle
01:46 : draw ellipse
01:49 : draw an image if you have an image file
01:52 : you can draw that under the screen
01:54 : can i see
01:56 : let's see these two layers are as input
01:58 : and output i mean the graphics layer is
02:01 : the output is whatever you want to
02:02 : render and the user event
02:06 : and event is
02:08 : very true yes so the graphics is output
02:11 : user event is input
02:13 : so
02:15 : um the graphics layer yeah has this kind
02:18 : of api where you fill a rectangle with
02:22 : green and then there's the rectangle
02:25 : that kind of thing and you can look up
02:27 : their api
02:29 : they have actually very good tutorials
02:31 : like this is a fill text allows you to
02:34 : draw text on the canvas
02:36 : and if you have an image
02:38 : you can make a new image and you can say
02:41 : draw the image at this location of the
02:43 : canvas
02:44 : qt has a similar api
02:47 : called the cue painter and i've been
02:49 : looking into it so it has very very
02:51 : similar methods
02:53 : draw a point draw a point draw a line
02:56 : draw a rectangle etc it is very similar
02:59 : in style to the canvas api so i call
03:03 : this the graphics layer if you are doing
03:06 : more low-level graphics there's opengl
03:10 : opengl is the standard for drawing
03:13 : graphics on a computer game developers
03:16 : tend to use this so this this is the
03:19 : lower level uh graphics
03:21 : api for the computer as in
03:24 : windows ships with opengl mac ships with
03:28 : opengl
03:29 : and opengl is an api that's designed to
03:33 : allow you to access the graphics
03:36 : hardware on your computer
03:39 : because a lot of computers nowadays ship
03:41 : with a graphics cpu a gpu
03:46 : which is a separate processor that's
03:49 : dedicated for
03:51 : doing calculations for graphics before
03:54 : they're drawn onto your screen so that
03:56 : everything can be faster like even if
03:59 : you have a lot of animations going on
04:01 : your screen
04:02 : they can be fast because there's a
04:04 : separate
04:05 : computer
04:06 : that's helping you do calculations for
04:09 : the graphics before it goes onto the
04:11 : screen and opengl
04:13 : gives you access to that graphics
04:15 : hardware it has a bit of a learning
04:17 : curve but um you know if you really care
04:20 : about
04:21 : rendering performance you need to know
04:23 : this so those are some
04:26 : examples of graphics layers uh in my
04:29 : case i use the terminal
04:32 : so my situation is quite different
04:35 : because it's all based on characters i'm
04:38 : just drawing characters on the terminal
04:40 : i can tell it to draw
04:43 : characters in certain color
04:45 : or with a certain background color and i
04:48 : can use
04:49 : weird unicode characters that help me to
04:52 : help me draw the things the way i want
04:54 : to draw and that's basically what i'm
04:57 : dealing with do you see like kind of
04:59 : each character as a pixel
05:02 : yes exactly i see a character as a pixel
05:05 : this pose is one of the best for
05:08 : learning terminal programming so an
05:10 : example is if you want to print the word
05:13 : hello world red then you just put this
05:16 : control code in front of it and this 31
05:20 : means red in addition to just color
05:22 : there control code to move your cursor
05:25 : anywhere on the screen so if you want to
05:28 : draw a box you just move this cursor to
05:31 : you move it to the top left corner and
05:33 : then you draw a line
05:35 : and then you move the cursor and draw
05:36 : the next parts of the box etc etc so
05:39 : that's what i'm dealing with as the
05:41 : graphics layer
05:44 : okay so
05:45 : the event is in canvas you're just using
05:48 : the dom event and then qt
05:51 : qt has an event system and you just
05:53 : learn qt's event system
05:56 : and
05:57 : opengl i'm not sure exactly how they do
06:00 : that
06:01 : but probably in each one there's a
06:04 : cross-bonding event library
06:07 : to
06:08 : like uh html events
06:12 : qt events
06:14 : i don't know how opengl does events
06:17 : in the terminal
06:19 : we have actually access to mouse events
06:22 : it's an extension of the terminal
06:25 : protocol i have to send a special
06:27 : control code i have this mouse on
06:29 : function and what it does is send this
06:33 : weird control code to the terminal
06:35 : and then that tells that
06:37 : or not mouse motion on i'm not using
06:40 : that there's also mouse motion which
06:42 : tracks your mouse move movement
06:45 : i'm actually not using that yet but the
06:47 : mouse on just
06:49 : tells the terminal when they click
06:51 : send me the click event i think that the
06:54 : first terminal that added this
06:56 : capability is x term so some of this
07:00 : stuff i learned from on npm there's a
07:04 : thing called terminal kit which is a
07:07 : which is a terminal based library that's
07:10 : written in node.js and i just looked at
07:13 : their source code
07:15 : and i learned a bunch of stuff from it
07:18 : so that's one thing i do and and then
07:21 : they
07:22 : they were nice to put in a lot of links
07:25 : to the resources that they used to build
07:28 : this thing
07:29 : so that linked linked me to other things
07:32 : that were helpful so x term is a
07:34 : protocol that supports some of these
07:37 : things and i could read more about how
07:39 : the protocols work one thing i would say
07:42 : is
07:43 : you if you're if you're planning to
07:45 : build
07:46 : like an ambitious abstraction
07:49 : like this you should get to learn your
07:53 : lower layer well you really want to
07:55 : learn how the lower layer works so since
07:58 : i feel like the canvas and the qt uh
08:02 : painter has a
08:04 : higher level apis
08:06 : uh because in terminal you don't have uh
08:09 : the api say draw a line or draw
08:12 : you you can draw a string you can
08:16 : right i mean in the terminal basically
08:18 : that's what you have you have print a
08:20 : string
08:21 : right
08:22 : and then if you want to print a string
08:24 : at a particular location like over here
08:27 : you say
08:29 : move the cursor over here and then print
08:31 : the string right so to me that's like
08:34 : more basic things yeah it's it's lower
08:36 : level but more rectangle stuff
08:39 : yeah
08:40 : it is it is so if you want to create
08:42 : like a terminal version of canvas you
08:45 : need to write all these methods yourself
08:47 : i didn't feel the need to that's true
08:50 : make an api that has these things i'll
08:53 : show you my canvas
08:55 : my my canvas is basically the region
09:00 : every widget draws things through the
09:03 : region
09:05 : and the region makes sure you don't draw
09:07 : out of the boundary that you're supposed
09:09 : to draw onto as well
09:11 : so it's got these tracks here
09:14 : but the the api of region the ones you
09:17 : use to draw are just draw
09:20 : and clear wrecked that's it
09:22 : these are the only drawing apis so you
09:25 : draw you draw a string at a particular
09:28 : position
09:29 : you can clear out a rectangle defined by
09:32 : the location of the rectangle and size
09:34 : of the rectangle that was enough for me
09:37 : to build everything i needed so far in
09:40 : the terminal it doesn't really make much
09:42 : sense to say draw a circle so everything
09:44 : has to be rectangles so i i think just
09:47 : because i have
09:49 : a lot of limitations of what i can do my
09:53 : job also ends up being simpler although
09:56 : i i did manage to do rounded rectangles
10:00 : and and i i did that using unicode
10:02 : character
10:04 : so do you have like a message to draw
10:06 : the rectangle is that like part of the
10:08 : api
10:10 : you have
10:11 : i just that's the code that draws a
10:13 : rounded rectangle
10:16 : and i just go through a loop and draw
10:19 : the little bar on the left side draw the
10:22 : little bar on the right side
10:24 : and then at the bottom
10:26 : there's a little rounded corner and a
10:28 : bar and another rounded corner
10:30 : nice i mean
10:32 : and there's not even that much code to
10:34 : do that right and then before i do that
10:36 : clear out the rectangle erase what's
10:39 : there before
10:40 : so yeah
10:42 : just just do it
10:43 : this could be extracted into a method
10:45 : but i just didn't feel the need to at
10:48 : this point
10:49 : it's sort of like we talked to jason
10:52 : about this like if you don't feel like
10:53 : it's painful
10:55 : then you don't need to extract anything
10:57 : i don't have a lot of situations where
10:59 : i'm drawing a border it's just a window
11:02 : class
11:03 : so that is that's the lower
11:06 : layer graphics and and so that shows you
11:10 : the
11:11 : problem you're up against i think it's
11:13 : like
11:14 : okay this is what i have to work with i
11:17 : can draw stuff i can draw shapes on the
11:19 : screen
11:20 : i can receive events and when it's a
11:23 : mouse even
11:25 : you'll receive x y coordinates right if
11:28 : if it's a keyboard even you receive like
11:30 : the key character or the key code
11:35 : so so so when you when you lay it out
11:37 : like this
11:38 : it shows you what you're up against
11:41 : these are the apis you're going to code
11:44 : against
11:45 : and then what api are you going to
11:47 : present to your consumer
11:50 : um
11:51 : that's something for you to design
11:54 : but we have a lot of examples out there
11:56 : that we can
11:58 : learn from
11:59 : and steal ideas from
12:01 : you can steal those ideas or or you can
12:04 : you know if you want to be adventurous
12:06 : you can say
12:07 : i want to do something completely
12:08 : different from what they are doing