How GUIs Work 3: Layout

In this episode 3 of How GUIs Works: I chat with Huiqi about how layouts in GUIs work. In particular, I describe my experience of first developing a homegrown layout algorithm, and then later switching to Flutter's layout scheme (I stole their algorithm by reimplementing it, not using any of their code). Here is a link to Flutter's layout algorithm: https://flutter.dev/docs/resources/architectural-overview#layout-and-rendering. Here is a link to their Google Talk https://www.youtube.com/watch?v=UUfXWzp0-DU. Also worth checking out is Jeremy Rose's article on this topic: https://blog.nornagon.net/ui-layout-algorithms/.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : okay so last time
00:03 : we talked about
00:04 : ui componentization
00:06 : so this is a to-do list you can see the
00:09 : visual hierarchy
00:11 : but in code we're gonna build it out
00:14 : as a
00:16 : code object hierarchy
00:18 : like at the top level there's an app and
00:21 : then we have a
00:23 : horizontal box and then there's a text
00:26 : field
00:27 : and then there's a
00:28 : button and then
00:30 : actually before the horizontal box
00:33 : there's a v box
00:35 : inside which there's a horizontal box
00:37 : and then maybe there's a list
00:40 : and then there's uh item
00:42 : [Music]
00:43 : item 2
00:45 : item 3. we built this tree to represent
00:49 : this
00:50 : visual hierarchy here what i want to
00:52 : talk about this time is
00:55 : how do you go from this tree of objects
00:58 : to
00:59 : this visual
01:01 : um that process is the layout and the
01:05 : paint first there's the
01:08 : layout and then there's the rendering so
01:11 : by layout you mean uh
01:13 : where you where do you draw stuff yeah
01:16 : well what is
01:17 : it yeah just imagine if i assign each
01:21 : node within the tree a position and a
01:25 : size each guy has a position
01:30 : is something
01:31 : and a size
01:32 : is something maybe
01:34 : size is 100 by
01:38 : 500. imagine i already gave you a layout
01:41 : engine
01:42 : and the layout engine will magically
01:45 : give
01:46 : each
01:47 : element
01:48 : a position and size
01:51 : with that information if i asked you to
01:55 : draw the ui
01:57 : it's actually relatively straightforward
02:00 : because
02:01 : because oh i need to draw app how do i
02:04 : do that
02:05 : well i know the position i know the size
02:07 : so i draw a rectangle well with rounded
02:10 : corners
02:11 : starting at this position
02:13 : uh with this size and that's true of
02:17 : many of these boxes that i've drawn most
02:19 : of them are just boxes
02:21 : right so you just draw a box at where
02:24 : you're supposed to draw the box
02:28 : with the text well
02:31 : just in addition to drawing the box you
02:33 : also draw some text
02:35 : how do you know where to draw like the
02:36 : text field we'll have to draw some text
02:39 : the button will have to draw some text
02:41 : as well calculate based on the font size
02:44 : of the text
02:45 : how tall does the text need to be and if
02:48 : you need to center it then calculate how
02:50 : long
02:52 : the string will end up being in that
02:54 : font size and then you try to center it
02:57 : within the box so there's some math
02:59 : involved
03:00 : but other than that it's kind of
03:02 : straightforward i mean if you've never
03:04 : done
03:05 : any kind of graphics programming like
03:07 : that where you have to deal with
03:09 : x y coordinates and stuff like that it
03:12 : takes a little bit of getting used to
03:13 : but once you
03:15 : are familiar with that it's
03:16 : straightforward so let's spend
03:19 : more time on the layout it's like how
03:21 : does the layout work how does a program
03:25 : magically figure out the position and
03:28 : size of each element on the screen it is
03:31 : a recursive
03:32 : tree walk
03:33 : if you looked at enough recursion
03:35 : problems pretty much whenever you see a
03:37 : tree you know it's recursion right so
03:40 : you walk from
03:41 : the top to the inside or do you walk
03:44 : from inside so like does the outside
03:47 : decide the inside or does the inside
03:49 : decide outside so initially i tried to
03:52 : come up with my own way to do this and
03:55 : it worked okay up to a point when i sort
03:58 : of got stuck trying to do scrolling and
04:01 : i did some research and i found a
04:04 : flutters layout algorithm i want to
04:06 : spend some time talking about their
04:08 : algorithm but
04:10 : before that let's just talk about the
04:12 : problem in general maybe just simplify
04:15 : this picture a little bit each box it's
04:18 : a component that will
04:20 : lay out its children
04:22 : horizontally right
04:24 : so this is an hbox if i add one
04:27 : component into it it would go here if i
04:30 : add another one it would go here
04:32 : if i add another one it would go here
04:34 : etc now in in the case of our ui we
04:38 : actually wanted the text field
04:40 : to stretch
04:44 : and then the button to take up the rest
04:46 : of the space in uh css
04:49 : nowadays people would normally use flex
04:51 : to do that
04:53 : in my framework i have a concept called
04:56 : stretch
04:57 : so i say this text
04:59 : field has stretch
05:01 : in the direction of x so so i would do
05:04 : something like when you add the child
05:06 : you say i want to add to the hbox the
05:10 : text field
05:11 : with a stretch in the x direction one
05:13 : observation is i want the h box
05:16 : to be responsible for the layout that's
05:19 : important because if it were a v box
05:22 : then i want the v box to do the layout
05:25 : in a different way
05:26 : right
05:27 : so i want the layout to be
05:30 : not a responsibility of the framework
05:33 : itself but the responsibility of the
05:35 : elements in particular the elements that
05:38 : act as containers and elements that are
05:41 : like text back text field and button
05:44 : they're not necessarily containers they
05:48 : will not need to worry about
05:50 : laying out children within them
05:52 : they still handled the layout of
05:55 : themselves
05:57 : at this point we could take that to mean
05:59 : decide
06:00 : what your position and size are going to
06:03 : be
06:04 : so the initial scheme that i did was
06:07 : each element would have a layout method
06:10 : it'll end up having some parameters but
06:12 : for now
06:13 : let's not worry about it if hbox was the
06:16 : root of your ui tree
06:18 : you're gonna lay out on the root
06:22 : of the tree because hbot is a container
06:25 : it it has the responsibility of going
06:28 : through each of his children and telling
06:31 : them to lay out
06:33 : only uh only containers have layout
06:37 : methods because i have to call layout i
06:40 : have to be able to call layout on my
06:41 : children as well and my children are not
06:44 : necessarily container so what do the
06:47 : non-container element layout do in my
06:50 : original algorithm it has to decide
06:53 : where its position and its size are
06:56 : but with this flutter algorithm it
06:59 : relaxed it and reduce it to decide what
07:02 : its size is only and then it diverts the
07:05 : position until later
07:08 : back to this example so after
07:10 : i call child.layout i should expect that
07:15 : dot size exists
07:17 : and has two numbers in it and i can i
07:20 : can receive the width and the height of
07:23 : child.size like this in python
07:27 : so so now now that i have the width and
07:30 : height of my child i can use that to
07:33 : calculate what the hbox's size should be
07:37 : based on what each child's size are but
07:41 : how do you know the stretch element size
07:44 : like in your example okay i probably can
07:46 : calculate how
07:48 : what's the size of the button based on
07:50 : the text
07:51 : but what
07:52 : left
07:53 : yeah the text because you don't know the
07:56 : parent size yet right exactly yeah it's
07:59 : a it's actually uh
08:01 : kind of tricky kind of tricky and i i
08:04 : will i will tell you about flutter's
08:06 : algorithm i also want to give a shout
08:08 : out to this guy jeremy rose because his
08:11 : article is how i learned about flutter's
08:14 : algorithm and this article was very
08:17 : helpful for me he said he researched a
08:19 : whole bunch of different layout
08:21 : algorithms in including the qt one
08:25 : the swift ui one
08:27 : and and css
08:31 : and he decided to go with flutter
08:33 : because it's
08:34 : the simplest one
08:36 : i found it flutters rendering pipeline
08:39 : by adam barth i just want to reference
08:42 : it because this is a main source of my
08:46 : research
08:47 : when you're laying out something that
08:49 : can contain some elements that are
08:51 : stretched and others that are not after
08:54 : i tell you is actually gonna seem
08:56 : extremely obvious oh wait wait i have to
08:59 : tell you the secret first though the
09:01 : secret is the layout function has a
09:04 : parameter which is called
09:06 : constraints okay
09:09 : and like what what's a constraint i will
09:12 : pretend it's typescript has a minimum
09:15 : width maximum width minimum height
09:19 : maximum height
09:20 : oh wow that's it that's constraints
09:23 : if you are an ui element like let's say
09:26 : you're a text field or you're a
09:30 : button um or let's say you're an hbox
09:34 : you're responsible for implementing your
09:36 : own layout
09:38 : and your layout method is going to
09:39 : receive the constraint which is just a
09:43 : set of four numbers okay all of these
09:45 : are optional so if if none of the
09:48 : numbers exist that means there are no
09:50 : constraints you you can you can choose
09:53 : whatever size you want for the layout
09:56 : phase all we're concerned about is
09:58 : coming up with your size which makes it
10:01 : easier
10:02 : when we render when we actually draw
10:05 : things onto the screen that's when we
10:07 : come up with what the position should be
10:09 : so the layout is one walk through the
10:11 : entire tree
10:13 : the rendering is another separate walk
10:16 : but the layout phase we're only
10:17 : concerned with the size so let's say
10:19 : you're the text field your layout method
10:22 : is called if you don't get any
10:24 : constraints at all all four of these
10:26 : numbers are not specified you get to
10:29 : choose what your own size is for a text
10:32 : field the obvious choice would be i'm
10:34 : just going to make myself as big as my
10:37 : text needs to be maybe my text is add in
10:41 : my case i i can't control the font size
10:44 : because i'm in the terminal then i'm
10:46 : just gonna return three characters
10:48 : across and one character vertical as my
10:50 : size if your pixel based graphics then
10:54 : based on the font size you calculate the
10:56 : size that is required to fit in this
10:59 : word so in this case because the
11:02 : recursion is from hbox to its children
11:04 : right so when you call hbox.layout you
11:07 : have the constraints which probably is
11:10 : the apps
11:11 : exactly initially we caught
11:14 : that layout with the constraint
11:17 : initially we want to use the size of the
11:20 : screen to constrain these things that's
11:22 : what you're thinking here so yeah you're
11:25 : thinking like we want to give the
11:27 : maximum width as the screen width and
11:29 : the maximum height as screen height yes
11:32 : right
11:33 : you're right you're right to assume that
11:35 : yes with just these four numbers you can
11:38 : either give a lot of freedom for your
11:40 : children to choose or very little
11:43 : freedom but the way you can give very
11:45 : little freedom for your children is by
11:47 : fixing all four numbers i want
11:50 : my
11:51 : child
11:52 : to be exactly
11:54 : 50 by 40 let's say
11:58 : then what i would do is child that
12:00 : layout min with 50
12:03 : max with
12:05 : 50 min with
12:07 : 40
12:08 : max that's 40 right
12:12 : so this is one of the strategies that
12:14 : you can use another strategy would be
12:17 : like hey i want my child to decide
12:21 : uh well you don't have to
12:23 : specify anything
12:25 : uh or you could or you could be lose
12:28 : with respect to you know the min minimum
12:31 : um
12:32 : then you're at at most
12:34 : um so you use a combination of these
12:37 : strategies to achieve the layout that
12:39 : you want so now coming back to the hbox
12:43 : layout where you have some stretched
12:46 : some elements stretched in the x
12:48 : direction so you want to do is first you
12:51 : pick out your non-stretched ones none
12:54 : stretched because like you do some kind
12:56 : of filter filter your children in some
12:59 : way to to get the non-stretched elements
13:02 : that you have and then you ask each of
13:04 : them
13:05 : what size do they want to be
13:08 : for child or non-stretched uh you you
13:12 : you say layout and you you're very
13:14 : relaxed about what size they can take so
13:18 : you just don't give them anything you
13:19 : just like
13:20 : you just tell me your best
13:22 : guess as to what your size is
13:24 : and then after you lay out i i'm gonna
13:27 : have your size
13:28 : and then i'm gonna get a child width and
13:31 : a child height after you lay out
13:33 : and then i'm gonna
13:35 : i'm gonna add them all up non-stretched
13:40 : total
13:42 : width right
13:43 : and i'm gonna add that
13:46 : so at the end of this loop i have the
13:48 : total
13:49 : width of all the non-stretch elements
13:52 : i'm gonna have the stretched ones as
13:54 : well
13:55 : and then now you can know what's the
13:58 : remaining space for the stretch yeah
14:01 : remaining is going to be like my
14:04 : constraints
14:05 : that max width minus
14:08 : the non-stretched total width and i i
14:10 : take this remaining and i spread it
14:13 : across the ones that are stretched so
14:16 : what if you have two that needs to
14:18 : stretch do they like each of them take
14:21 : yeah each of them take half yeah okay
14:24 : that's cool and you do something like
14:27 : with
14:28 : her
14:30 : equal remaining divided by
14:34 : the how how many stretched there are
14:37 : this is the width per stretch and then
14:39 : you go through them for child in stretch
14:42 : this time you use the other strategy
14:44 : which is to
14:46 : very strictly
14:47 : determine what their size should be
14:51 : and you say uh
14:53 : hey i want your min width
14:56 : to be that width i'm gonna specify and
14:58 : your max width is also going to be that
15:01 : okay gotcha does that does that make
15:04 : sense
15:06 : and and actually i i i sort of didn't
15:10 : think about the height but you also want
15:12 : to specify the height as well and but
15:15 : since we're in this case since we're
15:17 : laying things out horizontally and we're
15:20 : doing something like this right the
15:22 : height constraint i want to give each
15:24 : child is probably more or less the same
15:27 : height constraint that i got
15:29 : right
15:30 : because i want the child to be as tall
15:32 : as it can but
15:34 : but minus the padding basically it
15:36 : should be based on my own
15:39 : height constraint which would be like
15:41 : so i i i i pass on my height constraint
15:45 : to my children basically
15:47 : and also you know at the end i still
15:49 : have to come up with my own height and
15:51 : width so
15:53 : my width is going to be the sum of the
15:56 : widths of all of my children
15:58 : and my height is going to be
16:01 : the max of the height of all my children
16:05 : yeah that makes sense initialize it at
16:07 : zero so so once i get the
16:11 : width i'm just going to say my
16:13 : width
16:14 : plus equals to my child's width each
16:17 : time i get a child's width
16:20 : and i'll do the same here and then my
16:22 : height
16:23 : is going to be the
16:26 : max of
16:27 : the current my height and the child's
16:30 : height
16:31 : and and i'll also do that over here so
16:34 : at the end of both of these loops
16:36 : i have my height and my width and i can
16:39 : do something like hey my size is
16:42 : my width and my height
16:45 : so so now i have my size and that's how
16:47 : i calculate my size for an hbox layout
16:56 : nice
16:57 : that's cool
16:58 : yeah yeah so the key calculated on
17:01 : stretch once first yes and and and this
17:04 : this constraints trick is like it might
17:07 : seem like obvious but it was not obvious
17:10 : to me that just four numbers as the
17:13 : constraints is the only thing you need
17:16 : to to to be able to build very very
17:18 : flexible layouts
17:20 : and
17:21 : i think that's extremely cool yeah yeah
17:23 : i think this this makes sense because
17:26 : the complication before was like whether
17:28 : the children decide
17:31 : size versus parent design students says
17:34 : but this like makes it uh the
17:36 : constraints basically provide some sort
17:39 : of parameter of the parents and then
17:42 : still the parents is
17:44 : says
17:45 : finally is uh decided by the children
17:49 : so i like hands yeah
17:51 : you need communication both ways you
17:54 : need it's like a handshake kind of thing
17:57 : that the parents like oh i don't know
17:59 : what my size is until i know what my
18:02 : children's sides are but then the
18:04 : children are like uh well
18:07 : i am somewhat flexible if you give me
18:09 : more space i'll take it i think it's
18:12 : pretty remarkable that this scheme it's
18:14 : a very very simple rule set
18:17 : like all you the only rule is that every
18:21 : element in your ui tree must implement
18:24 : this layout method that takes in these
18:26 : constraints which are these phone
18:28 : numbers
18:29 : and it has to
18:30 : it has to respect the constraints like
18:33 : if you're told your max height is this
18:35 : you're not allowed to set your size to
18:37 : more than that
18:38 : but each number is optional so if
18:40 : they're not there you there's no
18:42 : constraint you can make your size
18:44 : infinite size if you want if and with
18:46 : just that
18:47 : you can create very complex layouts um
18:51 : at least that's what the flutter team
18:53 : told me they mainly focus on mobile apps
18:56 : it basically shows many complex mobile
18:59 : app layout and they're able to achieve
19:01 : all of those with this just very simple
19:04 : scheme