Free Up Disk Space with Recursion

In this Algorithm Friyay I demonstrate how to use recursion to free up disk space for your hard drive and how similar it is to working with ASTs.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : we wrote a program that would read in
00:03 : this code and figure out what the unused
00:06 : variables are of this program
00:08 : and uh
00:10 : um it did that by it used a package
00:15 : called s prima to parse the javascript
00:18 : and then the result of the parse is an
00:20 : ast we had it spit out the ast into this
00:26 : large
00:27 : json object basically
00:29 : and then we wrote a
00:31 : recursive traversal function
00:34 : that can traverse
00:36 : all the ast notes in this tree
00:40 : and then
00:42 : we we also made like a generic
00:44 : traverse function which we can pass in a
00:47 : visitor function and customize what
00:50 : we're going to do with the traverse
00:52 : function and at the end
00:54 : we made a function that finds our
00:56 : variable declarations
00:58 : find made a function defined so variable
01:00 : references
01:02 : uh you know
01:03 : uh made them match and if the ones that
01:06 : are declared but not reference those are
01:08 : unused and that worked out pretty well
01:11 : um this time
01:13 : i thought i would do something seemingly
01:17 : very different
01:18 : but
01:19 : in reality very much the same
01:22 : which is
01:24 : we're gonna write a uh disk
01:27 : cleanup program um so
01:30 : so one time
01:32 : uh
01:32 : i
01:33 : i was looking at my disk usage and i
01:37 : realized a large percent of the used
01:40 : disk is due to my node modules
01:44 : folder in various projects i have and
01:48 : and i do a lot of little experiments so
01:51 : for each one i make it's going to be a
01:53 : node modules program and it end up
01:56 : building up a lot of
01:59 : just
02:00 : disk usage
02:01 : and i wanted to clean them up
02:03 : so um so how would you do that um let's
02:07 : let's try something here
02:09 : um
02:10 : this might actually be a fun thing to
02:12 : follow along at home
02:15 : um i i think i'm gonna have like
02:17 : some of you
02:18 : pair with me um but uh if you're if you
02:22 : want to follow along yourself
02:24 : uh you're welcome too as well it might
02:26 : be fun because everybody has different
02:29 : contents on their desk
02:31 : um
02:33 : and uh it might be fun to see like
02:37 : unleashing this on what you have
02:40 : um
02:41 : [Music]
02:42 : let me see so the first thing you're
02:44 : gonna do
02:45 : basically is oh i don't i'm gonna
02:48 : probably use my
02:51 : another folder oh i don't have other
02:54 : folder
02:56 : i'll probably use my play
02:57 : play
02:59 : ground folder
03:01 : um first of all anyone wanna pair with
03:03 : me
03:05 : let me ask that first
03:10 : no
03:13 : jeweler i'm taking a more passive rolled
03:16 : stakes
03:23 : yeah same i'm trying to multitask sorry
03:26 : let's see in the sprint
03:29 : okay that's fine that's fine okay i'll
03:32 : just
03:32 : do it um
03:35 : so
03:36 : um
03:38 : first let me
03:39 : make a little startup program so i'm
03:42 : gonna do
03:44 : process start arcs
03:46 : um in node.js you use this to take in
03:49 : command line arguments
03:51 : i think usually
03:54 : uh
03:56 : file path i'm gonna take in as a
03:58 : parameter a file path
04:01 : and then let me print it out let's see
04:03 : what that is
04:08 : okay
04:12 : so
04:13 : maybe i'll do like
04:15 : i have a playground directory
04:19 : which i put a lot of my experiments
04:23 : um okay it's actually the second
04:26 : argument that i want okay so
04:29 : so this is argument zero node is
04:32 : argument zero this cleanup gs is
04:34 : argument one and then
04:36 : this playground directory is argument
04:39 : two so i'm gonna have that passed in and
04:41 : then i'm going to use
04:44 : this file path and try to traverse
04:47 : the directories inside this playground
04:50 : um
04:53 : uh
04:54 : yeah let's go ahead and do that
04:58 : so the first function
05:01 : i'm gonna use
05:03 : is
05:06 : um
05:08 : is the reader
05:14 : it's the reader uh
05:16 : function
05:17 : and this lists the contents of the
05:19 : directory for you
05:21 : so i'm gonna do say
05:23 : and also i'm using the promises version
05:26 : of the file system
05:28 : so say
05:29 : fs.reader
05:32 : the given file path
05:37 : and then because this is the promises
05:39 : version of the library i'm going to
05:41 : outweight
05:43 : and then i'll end up with a bunch of
05:46 : entries
05:47 : remind me what um
05:49 : both argv is and why you're accessing
05:51 : the second or index two of that array
05:55 : oh yes
05:56 : uh i just
05:58 : okay
05:59 : uh
06:00 : let's complete this thought first so i
06:03 : just uh read the read this file
06:07 : directory
06:08 : and it gave me all the things that are
06:11 : in that directory oh so sorry to
06:14 : answer this question really quick
06:16 : so basically this
06:18 : is
06:19 : process that rv is
06:21 : of the
06:24 : is basically your command line
06:27 : command this guy
06:29 : node disk cleanup.js
06:32 : users airport home playground
06:36 : uh it's a list of these
06:38 : strings
06:40 : of your command is what it is so if i
06:42 : print this out
06:54 : uh this is what you see oh actually it
06:56 : has resolved node to the specific path
07:00 : for node
07:01 : it has also resolved the um
07:04 : location of my js script
07:09 : does that make sense
07:10 : so so i'm just grabbing the third one
07:13 : because the third one is the parameter
07:15 : into my program
07:18 : and
07:19 : i i should say the second one if we
07:22 : we use zero indexing
07:25 : and what were zero and one again sorry
07:27 : zero would have been that
07:31 : the location of node
07:33 : and 1 would have been the location of my
07:36 : program
07:38 : oh okay
07:40 : yeah
07:41 : so i'm not mistaken that's consistent
07:43 : across um
07:45 : cli interfaces like the first
07:47 : argument is always the uh
07:50 : the path to the program that you're
07:51 : running
07:52 : yeah it's slightly confusing sometimes
07:56 : because
07:59 : like in some
08:00 : languages it would be
08:03 : uh
08:05 : it would be shifted like
08:07 : one would be the name of their program
08:11 : because it's there's no node
08:14 : like if it's a c program
08:16 : then the first thing in the list would
08:19 : just be the program itself right there's
08:22 : no other thing that you use to run the c
08:25 : program it's a binary already
08:28 : so but yeah for for node.js i think
08:32 : this this is consistent
08:38 : okay
08:39 : so now no no so this is the way i'm
08:42 : gonna use oh and often when i write code
08:45 : like this i would write hey if you
08:47 : didn't provide a file path
08:52 : i'm gonna print hate
08:57 : please provide a file path
09:01 : so that if you forget that and
09:04 : it will tell you that
09:06 : um okay now
09:09 : now basically uh what i really want to
09:11 : do is
09:13 : okay i i printed out the entries
09:17 : of this directory but i actually want to
09:19 : recurse so as a first task
09:22 : let's recurse through
09:26 : oh
09:27 : nested files
09:29 : maybe maybe i don't do it to everything
09:31 : because there's quite a lot of stuff in
09:33 : there
09:34 : so if i do it to lively kernel or
09:37 : something
09:38 : uh so this is everything in lively
09:41 : kernel but we want to recurse through
09:45 : uh if it's a directory we want to drill
09:48 : down so again this is
09:52 : where we use recursion um
09:56 : so i'm gonna write the traverse function
09:59 : um where
10:00 : where's the starting point of the
10:03 : traversal is going to be some file path
10:08 : and i'm going to make this one an async
10:10 : function because
10:13 : the mechanism of the traversal which is
10:16 : this reader function is async
10:19 : so if i want to be able to use reader
10:22 : inside my traversal function i have to
10:24 : make my traverse function
10:26 : async as well
10:29 : okay so i'm gonna actually move this in
10:31 : here
10:32 : and say read the entries
10:35 : and then now
10:37 : once i have these entries
10:39 : uh i'm gonna for
10:45 : each entry of entries
10:49 : um
10:50 : i'm going to make the entry path
10:55 : i think
10:56 : in note you're supposed to do it with
10:59 : path dot join
11:01 : so i tap that join my file path
11:06 : and then entry
11:07 : and then that would give me the entry
11:10 : path i'm
11:11 : connecting this
11:13 : to that basically
11:15 : so that's the entry path and then i may
11:18 : traverse or i may not
11:21 : uh how do i know if i want to
11:24 : traverse
11:26 : into the entry path
11:32 : you want to check whether it's a
11:34 : directory right
11:35 : yeah yeah that's exactly i want to check
11:38 : whether it's a directory how do you
11:39 : check if something is a directory or not
11:42 : oh there's this stat
11:45 : function
11:47 : yeah it's there stat
11:49 : because that give it a path it'll give
11:52 : you back as that
11:54 : object
11:55 : what is a stats object well stats object
11:58 : is a object that has all these things on
12:01 : it
12:02 : uh including the i note number uh
12:07 : i'm not going to talk about inode but uh
12:10 : it does have like the modification time
12:12 : creation time access time
12:15 : uh
12:16 : so that's useful actually if you're
12:18 : making a build system or something
12:21 : but what it also has is these methods
12:25 : including is it a directory
12:27 : and is it a file
12:29 : so that's what i'm going to use so i'm
12:31 : going to say hey can we get the entry
12:33 : stacked
12:37 : do i have a higher level question about
12:40 : stuff especially like seeing some of the
12:42 : properties on that stats object
12:45 : like
12:46 : in your experience is this
12:48 : like the way that node
12:49 : interacts with the file system is this
12:51 : like pretty standard if you were to use
12:54 : a different language to do that or maybe
12:55 : even like a script um like a shell
12:58 : language
12:59 : um like these properties and stuff is
13:02 : this like pretty standard uh
13:04 : between different areas
13:07 : um
13:08 : so
13:10 : well
13:11 : yes
13:12 : it's
13:13 : um
13:14 : so there's difference in terms of
13:17 : like different operating systems have
13:19 : slightly different terminology and maybe
13:21 : even functionality but
13:24 : node kind of had to normalize it across
13:27 : different operating systems because
13:29 : that's one of node's selling points it
13:31 : can run on
13:33 : the main all the different main
13:35 : operating systems
13:39 : i know
13:40 : i thought i node was a unix terminology
13:44 : but i'm sure windows has something
13:46 : that's equivalent to it is essentially
13:49 : the identifier
13:50 : for that thing
13:52 : whether it's a file or a directory uh
13:55 : there has to be an identifier
13:58 : in the system that tells the system oh
14:00 : this means that specific
14:04 : thing in the file system
14:06 : uh block size
14:09 : uh let's not get into that
14:11 : that could take a while but yeah yeah
14:13 : these access time creation time etc last
14:17 : access last modified
14:19 : time of creation that's exists in all
14:22 : the operating systems so we can do that
14:24 : i don't know what that is i don't know
14:26 : it's def
14:27 : and uh yeah size
14:30 : that's the size of a file
14:33 : i don't actually know what the size of a
14:35 : directory means
14:37 : uh
14:39 : i'm sure it takes some uh
14:42 : space to store a directory node on the
14:45 : file system
14:47 : uh but i don't know what that is maybe
14:49 : we can check that out later
14:51 : but
14:52 : uh but yeah this this has been
14:54 : normalized across the different systems
14:57 : and they work on all three
14:59 : um
15:00 : so now what i'm going to do is i'm going
15:02 : to ask hey is
15:04 : is this a directory
15:09 : or
15:11 : is this a file
15:17 : and what if it's neither of those
15:20 : i don't know what that means
15:22 : but i'm not gonna worry about it
15:27 : i i haven't came up with i came across a
15:31 : case where it's neither of those
15:33 : i'm just gonna throw an error and not
15:35 : worry about it and if i'm wrong about
15:37 : that
15:38 : i will get this error
15:41 : um
15:43 : okay so if it's a file
15:46 : what should we do if it's a directory
15:48 : what should we do
15:53 : if it's a uh directory
15:55 : it's recursion time
15:57 : yeah yeah let's recurse that means call
16:00 : ourselves
16:02 : and we're gonna have to wait because we
16:04 : are async
16:06 : so
16:08 : um
16:09 : and we have the entry path already handy
16:13 : so that's nice
16:15 : are we even returning anything i don't
16:18 : know that we are
16:20 : um and then if it's a file
16:23 : what should we do should we do anything
16:26 : even
16:27 : um
16:29 : we probably at some point we want to
16:31 : introduce this visit function
16:33 : exactly like we did last time let's go
16:36 : ahead and do that
16:38 : and
16:41 : and then similar to last time we're just
16:43 : gonna call visit right here
16:46 : um
16:48 : let me think
16:53 : one thing i'm thinking about is it would
16:55 : be useful
16:57 : like you would call this like
17:00 : traverse
17:03 : uh to pass in file path and i'm gonna
17:06 : make a little
17:07 : visit function
17:09 : that's gonna
17:10 : um
17:11 : receive the file path like this
17:16 : well
17:17 : you know what let's just just do this
17:19 : first get this much working so let's say
17:21 : i just print out the file path visiting
17:28 : okay so we're gonna have this visit
17:30 : function it's gonna visit
17:32 : every
17:33 : file or directory within this
17:36 : subdirectory
17:39 : do i have to do anything if it's a file
17:43 : to question
17:50 : what's the goal of the program
17:53 : oh the way um for now we're for now
17:56 : we're just gonna write a generic
17:58 : traverse function
18:00 : but the the overall goal of the
18:04 : program is
18:06 : original i actually haven't said what it
18:09 : was but
18:10 : my original goal was
18:12 : i want to traverse
18:14 : into my
18:16 : uh
18:17 : my
18:18 : subdirectory where i have a lot of
18:21 : programming projects
18:22 : and i want to find all of the
18:25 : see if any of them have a node modules
18:29 : directory in there and i want to see if
18:32 : i can delete them and my criteria was
18:36 : if i haven't touched that project in
18:39 : over two months i'm going to delete that
18:41 : no modules folder
18:45 : nice okay that's cool
18:47 : does that sound like a plan like that's
18:49 : a very practical you might actually
18:51 : wanna
18:53 : you might and when i did this actually i
18:55 : saved like 40 gigs
18:59 : yeah i would say i mean a great first
19:01 : step always is printing something
19:03 : yeah but for now we're just writing a
19:06 : generic traverse function exactly like
19:09 : we did last time with the ast
19:11 : and yeah let's just print something and
19:13 : let's just try this see if this works
19:16 : you know i'm just gonna do it inside
19:18 : this sub directory
19:23 : visit is not a function oh i forgot to
19:26 : pass visit
19:28 : to my uh
19:30 : to my recursive call i think
19:36 : there
19:39 : okay
19:40 : uh
19:41 : so i i started in the lively kernel
19:44 : directory
19:46 : and it traversed
19:50 : uh
19:52 : oh it actually did not
19:54 : print the files
19:59 : uh i guess we should trigger even if
20:01 : it's a file
20:03 : i should call traverse on it just
20:04 : because just to get visit to be called
20:08 : right
20:10 : yeah
20:13 : or maybe just go visit on these
20:16 : entries
20:19 : on the
20:20 : uh
20:21 : entry path here
20:25 : that seems reasonable
20:27 : oh wait wait wait wait i i i'm having
20:29 : second thoughts actually about this
20:32 : structure because
20:34 : should we even bother
20:36 : doing reader
20:39 : because we don't even know if it's a
20:40 : directory here
20:42 : right
20:44 : we should only read there
20:46 : if it's a directory
20:49 : right
20:52 : does that make sense
20:54 : yeah for sure so you want to move that
20:56 : entry stat line yeah so i think what i
20:59 : should do is
21:00 : yeah this this whole thing should
21:03 : actually be inside of that
21:06 : and but which means i have to do a stat
21:09 : on this guy
21:11 : so
21:12 : just just to make it less confusing for
21:15 : you i'm gonna start over
21:17 : but we can use bits and pieces from here
21:20 : um
21:22 : which also by the way uh it's it's
21:24 : probably good advice in many cases like
21:28 : if if if you're the the mess that you
21:31 : made is confusing you
21:33 : you probably just want to ignore it and
21:35 : start over
21:37 : um
21:38 : so what i'm going to do is do the stat
21:41 : thing
21:41 : first but not to one of the entries i'm
21:45 : gonna do it to
21:46 : myself
21:48 : and then i'll
21:50 : get my stat
21:53 : um and then
21:55 : oh what is nice is now i can actually
21:58 : give the stat
21:59 : to the visit function because that might
22:01 : be useful for the visit function
22:07 : but now i can say
22:09 : hey if
22:11 : i am a directory
22:18 : then
22:19 : i want to read there
22:25 : um
22:28 : and then
22:29 : i'm gonna
22:32 : traverse
22:36 : wait i'm gonna do this for loop
22:43 : uh generate this entry pass
22:52 : traverse the entry pass pass in the
22:54 : visit function oh wait
22:59 : um
23:00 : [Music]
23:03 : that actually looks pretty good
23:06 : um
23:08 : if do i even care to check if it's a
23:12 : file
23:14 : is file path and the traverse function
23:17 : already joined or is it just a fragment
23:23 : uh
23:24 : where
23:26 : that's that parameter that you have
23:27 : highlighted
23:29 : oh this one
23:30 : oh yeah it's already joined
23:32 : here
23:33 : gotcha cool okay that's a mistake i make
23:35 : all the time so
23:38 : yeah
23:39 : i actually think there's nothing to do
23:43 : if it's a file or not and even on the
23:46 : else clause
23:48 : there's also nothing to do i mean i
23:50 : could throw this error
23:52 : but or i don't have to either
23:55 : yeah i actually think there's nothing to
23:57 : do if it's not a directory
24:00 : so
24:02 : um
24:03 : let's see if this works
24:06 : yep
24:08 : uh
24:08 : [Music]
24:11 : yeah that's looking good for me
24:20 : there's a lot of files in this lively
24:23 : kernel project
24:24 : um
24:26 : this is not my project it's dan angus's
24:30 : javascript clone or small talk or
24:32 : something
24:33 : um
24:35 : so okay i think this traverse function
24:37 : works it even drills into the git
24:39 : directory
24:40 : uh which begs another question we
24:42 : probably also you remember last time we
24:44 : said hey you can return false
24:48 : from your visit function
24:50 : and then we won't traverse further down
24:52 : that node we can also use that and what
24:55 : we can use that for is to ignore
24:58 : directories like the git directory right
25:01 : and say hey if this directory name is
25:04 : dot git
25:05 : return false and then it won't traverse
25:08 : down into the git directory right
25:11 : so that's useful
25:14 : so what i'm going to do is
25:18 : [Music]
25:20 : i think there's a function in node.js
25:23 : called phase name
25:25 : which we can use
25:27 : which just means
25:32 : i think it's under a path
25:34 : oh yeah yeah yeah thank you
25:36 : that's why i couldn't find
25:39 : path
25:41 : there
25:43 : all right
25:45 : so base name is gonna return you
25:48 : like the name of your file
25:51 : discarding the directory portion of your
25:54 : file path basically so
25:56 : um what would i well the reason i that
25:59 : would be useful is i want to know if the
26:01 : base name is dot get here
26:04 : so let's do that so i pass that base
26:07 : name
26:08 : a file path let's say
26:11 : hey if that is
26:14 : that get
26:16 : return false and then then it i wanted
26:20 : to stop the
26:22 : drill down of that directory
26:25 : um otherwise i'm not going to return
26:27 : false
26:28 : so now i can say hey if
26:35 : if you return false
26:38 : i'm just gonna
26:39 : early return here
26:45 : okay so it went through everything
26:51 : yeah but it did not go into the get the
26:55 : dot get directory so that worked
26:58 : [Music]
27:00 : yeah
27:01 : we do this
27:03 : okay so now we have this uh generic
27:06 : traverse function
27:08 : i think it works
27:09 : i think from my brief testing i think it
27:12 : works
27:13 : we can use it to do a variety of
27:15 : different things and
27:17 : and that's very powerful just like last
27:20 : time
27:21 : with the ast we use the traverse
27:23 : function to do two different things
27:26 : this time we can use this same traverse
27:28 : function to do a variety of different
27:30 : things
27:31 : um so one thing you might want to do is
27:35 : hey i want to calculate
27:38 : the disk usage
27:40 : under this directory
27:43 : so the way you can do that is like
27:46 : like i'm going to make a
27:49 : calc
27:50 : disc
27:51 : usage
27:56 : um
27:57 : let me try that
28:01 : so calculate this usage i'm going to
28:04 : traverse
28:09 : and i can use an arrow function
28:13 : so remember the stat function
28:17 : has a
28:18 : size
28:19 : property on it i don't know if you
28:21 : remember that
28:23 : uh f s stats uh
28:28 : state stat
28:31 : stat
28:32 : fstat
28:34 : yeah it has a size property
28:37 : and i'm gonna do this
28:39 : stat size
28:41 : i'm gonna await it
28:43 : because it's async
28:45 : and i'm gonna have a
28:47 : total size
28:49 : out here
28:51 : and i'm just gonna add to it
28:54 : and
28:55 : return total size
28:58 : and as assuming this visit function will
29:01 : go through every single
29:03 : node
29:05 : in the subtree
29:06 : will end up with the total size here
29:09 : so let's try
29:10 : this uh we might also
29:14 : maybe print out the size of each node
29:17 : that might be a good diagnostic
29:20 : thing
29:32 : okay so i'll say
29:56 : oh
29:57 : i forgot to await
30:03 : okay
30:06 : that's cool what
30:08 : is the unit
30:11 : of the size
30:13 : just bite
30:14 : i guess yeah yeah i think it's just bite
30:17 : so
30:18 : that means this directory contains 89
30:22 : megabytes
30:24 : worth of stuff
30:26 : that's substantial
30:30 : but
30:31 : it would also be cool to like
30:34 : sort rank it and see what the large
30:36 : files are for example
30:39 : uh that would be a cool thing to do
30:41 : um
30:43 : i wonder also whether
30:46 : yeah what's the size of a directory i
30:48 : think that's a directory so the size of
30:51 : this directory is 160.
30:54 : does it
30:55 : matter
30:58 : why do directories have different sizes
31:01 : oh maybe because they have different
31:03 : number of entries
31:05 : maybe they store the entries inside the
31:08 : directories
31:10 : does that sound right reasonable
31:13 : does anybody know
31:17 : like they're the ones that have more
31:20 : things in them
31:23 : as direct children are probably going to
31:26 : be
31:28 : larger directories that's my guess but
31:32 : yeah so we have a way to like ask hey
31:37 : uh how how much is this eating up how
31:40 : much disc space is this eating up
31:43 : um
31:45 : so that's cool
31:46 : uh
31:48 : now
31:49 : we might
31:50 : the next thing we might do is like hey
31:53 : well i wanna i'm gonna look at
31:55 : node modules
31:58 : directories and so i know i happen to
32:00 : know
32:02 : in my
32:03 : playground
32:06 : directory
32:08 : i probably have like
32:10 : a lot of
32:11 : angular
32:13 : projects test projects
32:16 : um
32:18 : this one
32:19 : may not even have a
32:22 : node modules folder
32:24 : how this one does this one has a node
32:26 : modules folder so if if we drill it into
32:29 : this one
32:37 : we drew in there
32:41 : wow
32:46 : first of all it got so much stuff in it
32:49 : that is taking a long time just to run
32:52 : this traversal
32:55 : now the printing out of stuff
32:57 : let's slow it down a little bit
32:59 : and then doing the stat on every single
33:02 : file
33:04 : also slows it down a little bit
33:07 : but yeah this there's a whole bunch of
33:09 : stuff in here
33:11 : um
33:13 : what i want to know though oh there it
33:15 : is so there's
33:18 : 224 megabytes of stuff
33:21 : uh what i want to do though what i want
33:23 : to know though is
33:25 : how much space is the node modules
33:29 : folder taking up
33:31 : so
33:32 : how can i
33:33 : get an answer to that using the code we
33:36 : already have
33:42 : i basically i
33:44 : given given that given this directory i
33:47 : want to find
33:49 : any node modules folder
33:52 : inside it and then i want to be able to
33:54 : tell
33:56 : how much disk space is that node modules
33:59 : folder consuming
34:02 : you would want to
34:05 : create a new visitor
34:07 : that
34:09 : basically accumulates
34:11 : file sizes
34:12 : and once you get to
34:15 : the folder that's named node modules you
34:18 : uh
34:20 : don't enter it with the original visitor
34:23 : but send in
34:25 : the
34:26 : you know file size measuring
34:28 : visitor yeah that one
34:31 : yeah we already have it we have already
34:33 : have this yeah you call it this guy and
34:36 : then i'll send that in
34:38 : uh yeah so we we do a traversal
34:41 : let's see
34:43 : okay i'm gonna do what you said so the
34:45 : search is gonna traverse
34:50 : on the given path
34:58 : and it's going to get a stat
35:01 : okay
35:02 : and then i want to tell be able to tell
35:04 : is this a node modules folder so what i
35:08 : can do is if
35:10 : stat
35:11 : is directory
35:14 : and
35:16 : path
35:17 : base name
35:19 : of the file path
35:22 : is called
35:23 : node modules
35:26 : i do something to it uh what do i do
35:33 : danny
35:34 : what did i do to it
35:36 : uh
35:38 : this is when you would traverse it with
35:40 : your kelp disk usage
35:45 : yeah i just call it characters usage
35:50 : yeah i probably want to await that
35:55 : uh and then what
36:01 : this is tricky
36:05 : you need some way to
36:08 : to
36:08 : recursively accumulate that total size
36:11 : right
36:12 : because you don't know how
36:14 : deep the
36:15 : file tree is going to go and you need to
36:19 : keep that
36:20 : number alive in between the recursive
36:23 : function called um but you also don't
36:26 : necessarily want to have to add a
36:29 : parameter to your function because then
36:33 : like you have like a non-standard
36:34 : interface
36:36 : wait a minute
36:38 : um i'm not totally following you i need
36:42 : to keep
36:44 : um
36:45 : what you think you need to accumulate
36:50 : yeah so you need to not just
36:53 : measure the files that are in the root
36:56 : of of your node modules directory you
36:59 : need to measure files in subfolders and
37:02 : subfolders and subfolders and so on
37:05 : oh but basically basically
37:07 : basically the note modules folder right
37:11 : right
37:12 : oh okay uh i confused matters by calling
37:15 : this guy file path same as shadowing
37:18 : that one
37:19 : so
37:21 : but um i could just say
37:24 : child or a file path maybe
37:29 : so it's different
37:31 : um i think that's actually okay because
37:34 : if i'd if i recall calc disk usage
37:38 : don't forget to change it online 19 toby
37:41 : before you forget
37:43 : thank you very much yeah i've
37:46 : been confused okay so if i if i'd call
37:49 : call this
37:51 : calc disk usage on this a file path this
37:54 : file path corresponds to the node
37:56 : modules folder
37:58 : and i'm i'm calculating the disk usage
38:02 : only for this stuff inside the no
38:04 : modules folder
38:06 : i think that's okay
38:07 : that doesn't
38:09 : confuse it with other stuff
38:11 : that's in my top level directory
38:15 : does that make sense
38:17 : but there's one thing i want to do which
38:19 : is
38:20 : uh
38:21 : stop the top level traversal from
38:23 : drilling in
38:26 : into the no modules folder because i
38:29 : have a separate function that does that
38:37 : right
38:41 : does that make sense
38:43 : and i think
38:44 : i think this is fine
38:47 : i think this works
38:48 : but
38:50 : can you scroll down just a minute i
38:52 : might be
38:53 : missing something
38:59 : yeah i think because you're not
39:01 : recursively calling that arrow function
39:03 : on line 32
39:06 : um
39:08 : it's only going to be able to measure
39:12 : uh
39:13 : the
39:14 : top level directories in node modules
39:17 : right but don't you have to actually
39:18 : like dive into each of the subfolders
39:21 : and like measure each of the
39:24 : files
39:25 : that will already be done because
39:28 : the traverse function does that
39:31 : oops
39:32 : the traverse function
39:36 : if implemented correctly
39:38 : should guarantee that this
39:40 : visit function gets to visit
39:44 : every
39:45 : node within the subtree
39:47 : gosha okay that's the part i was missing
39:50 : sorry
39:52 : yeah it's a little
39:53 : weird like switching back i think the
39:56 : mentality
39:59 : you you like when you're using the visit
40:02 : function
40:03 : you just need to assume
40:05 : the traversal is always going to be done
40:08 : for you whereas if you're traversing
40:10 : yourself then that's not the case you
40:12 : have to call you have to do the
40:14 : recursion
40:15 : but basically we
40:17 : by by making this visit function work
40:21 : we have to increase the level of
40:23 : abstraction
40:25 : such that the user of the traverse
40:27 : function doesn't have to worry about
40:28 : recursion
40:31 : yeah that totally makes sense
40:35 : so yeah so what's happening in here and
40:38 : and also if anybody
40:40 : wants more clarification i i'll try my
40:43 : best now just ask me more questions
40:46 : um but the way this works is we're gonna
40:49 : traver we're gonna start at this top
40:52 : level directory
40:54 : i i have a project called ng
40:57 : 2a binding
40:58 : i'm going to dive into it and i'm going
41:01 : to assume my visitor is going to visit
41:04 : every single node in this in the
41:06 : directory tree
41:08 : uh
41:09 : in a depth-first manner but i can
41:13 : when i hit a directory and i want to
41:16 : disable the traversal of of this
41:19 : traverse function i return false and
41:22 : then so i i'm telling the top level
41:25 : traverse function don't go any deeper if
41:28 : you see a node modules folder
41:31 : but then here i'm calling another
41:33 : traversal i'm starting another traversal
41:37 : that does go into the no modules folder
41:39 : myself
41:40 : to calculate this usage um
41:43 : and then the
41:44 : the if it's if it's anything else i'll
41:47 : just let the main traversal function do
41:49 : its thing that way it'll
41:52 : find if it there's like a nested nested
41:54 : folder that has another node modules
41:56 : folder it'll be able to find it
42:00 : so yeah let's
42:01 : see if this works um so here i'm gonna
42:04 : say
42:07 : found
42:10 : let me do this syntax found
42:15 : no not just folder
42:17 : that
42:22 : which uses
42:27 : [Music]
42:30 : five
42:42 : [Music]
42:44 : oh
42:45 : i'm gonna need the visitor to be async
42:49 : as well
42:56 : which is fine that's allowed
43:00 : okay maybe
43:17 : okay
43:17 : it's going
43:20 : i don't know how long it's gonna take
43:22 : oh not too bad
43:24 : okay yeah so we found this no modules
43:27 : which uses that much space
43:31 : um
43:32 : that's cool if
43:35 : i wonder
43:36 : if i do it at the top the parent
43:39 : directory it's actually going to go into
43:41 : a lot of little projects i have each one
43:46 : might have a note modules folder
43:48 : within it
43:49 : and
43:50 : it will end up
43:54 : oh
43:57 : what's going
44:07 : you have on sim links happening
44:09 : there might be a sim link yes
44:12 : um
44:13 : [Music]
44:14 : i think yes because
44:19 : power bi js
44:23 : yeah that that might be a sim link this
44:25 : is like infinite
44:27 : oh no wait
44:29 : it eventually
44:32 : goes into it index
44:37 : this does look
44:38 : very much like a sim link though
44:46 : so
44:47 : there's a way
44:49 : when you do a yeah
44:53 : there's a way to tell it to not
44:56 : follow assembling
44:58 : and is it this one
45:06 : oh it might be
45:07 : instead of using stat
45:10 : i might i should maybe i should use else
45:12 : that
45:14 : uh let me see
45:16 : retrieve this staff for the symbolic
45:18 : link
45:19 : where status is
45:21 : else that is identical to stat except
45:24 : that if the path is a symbolic link then
45:27 : the link itself
45:29 : is static not the file that it refers to
45:32 : yeah so i don't want it to follow the
45:34 : link maybe that'll help
45:36 : oops
45:43 : yeah that did fix the problem
45:46 : so now it's it's going through all my
45:49 : projects and telling me
45:51 : if it's got a no modules folder
45:54 : and if it does
45:56 : how much disk space
45:57 : it consumes
46:01 : yeah so that that seems to work pretty
46:04 : well
46:07 : um okay
46:09 : so
46:12 : so um
46:13 : um i guess the next thing i could do
46:17 : is
46:19 : i could maybe ask
46:22 : um
46:25 : maybe i could ask like hey how when was
46:28 : this project
46:30 : last touched
46:33 : and uh how can i do that
46:35 : let me see
46:37 : um
46:39 : i think a just a very basic way to do it
46:44 : is just go through its files and ask
46:47 : it's
46:48 : modified time
46:50 : um
46:51 : yeah let's just do that
46:53 : so
46:57 : get
46:59 : project last modified
47:04 : and given that
47:05 : project directory okay
47:08 : so
47:08 : again i'm going to use a traverse
47:15 : um
47:15 : [Music]
47:19 : i'll call this a file path
47:25 : um let's skip let's skip the note
47:27 : modules folder select if
47:31 : path based name of a file path if that
47:35 : is a node modules
47:40 : just skip it
47:44 : because i want to look at the other
47:46 : files
47:47 : but otherwise we can get the stat dot m
47:51 : time which would give us the
47:54 : modified time of that file
47:57 : or directory
47:59 : oh i forgot there's a syntax error here
48:01 : okay
48:02 : so then
48:03 : uh what i could do is just do the
48:07 : sort of uh
48:09 : the oldest modified time out of all the
48:12 : files
48:13 : so i can do
48:16 : last modified
48:19 : uh well initialize it with like a
48:22 : null or something
48:23 : but
48:25 : um
48:26 : maybe just do if it's a file if
48:29 : that is file then i'll do this
48:33 : so if it's a file then i want to see
48:37 : if
48:39 : uh if less modified
48:42 : is not declared i'll say last modified
48:46 : is this file's
48:49 : last modified time
48:51 : else
48:53 : else if
48:58 : this file's modified time is greater
49:00 : than last modified
49:04 : then we will use that as the last
49:06 : modified time
49:12 : and then at after
49:14 : all of that assuming all this is
49:17 : happening sequentially that's why we
49:19 : need to await
49:20 : at the end we should have the last
49:23 : of the
49:25 : latest last modified time out of all the
49:28 : files that are not in the note modules
49:29 : folder
49:31 : um does that make sense
49:34 : okay so
49:35 : maybe what i'll do here is to say
49:39 : [Music]
49:40 : uh
49:42 : oh let me just test this function
50:00 : maybe maybe
50:01 : do like
50:07 : just test just what this one function
50:10 : for now
50:35 : so maybe like the angie something
50:42 : okay
50:43 : you gave me a last modified date which
50:45 : is 2017
50:48 : okay
50:53 : 2018 okay
50:55 : that sounds reasonable to me
50:57 : so
50:58 : so i have a function that can tell me
51:00 : the last modified time
51:03 : uh
51:04 : of a
51:07 : subdirectory
51:09 : basically
51:10 : um
51:12 : and okay
51:14 : so now
51:16 : what we could do is we could put
51:18 : everything together i think
51:20 : and say hey
51:24 : if i get a no modules folder
51:28 : i want to find the
51:30 : last modified time
51:33 : of the parent directory because the
51:36 : parent directory is the project folder
51:40 : we'll just assume that
51:47 : so
51:50 : so i'm gonna do like the way the way you
51:52 : can get the
51:54 : parent folder is
51:57 : their name so parenter is path that
52:00 : their name of
52:03 : of this path
52:04 : so a file path
52:07 : so this one will give you the parent
52:09 : directory of the node modules folder
52:12 : and then we'll get when was this
52:15 : the content last modified
52:18 : and
52:19 : [Music]
52:26 : and then we'll print it out
52:29 : let's try that
52:42 : yeah so we have something like hey i
52:44 : found that no modules folder it uses
52:47 : that many bytes of disk usage and its
52:49 : parent
52:50 : directory
52:52 : was last modified on that date
52:55 : and so forth
53:10 : okay
53:11 : uh
53:14 : okay and then we're almost there so like
53:17 : if the only
53:19 : thing left to do is like if if we want
53:22 : to actually delete
53:24 : that folder
53:25 : i could issue a command like exec
53:29 : rm
53:31 : r
53:33 : then the name of the node modules filter
53:38 : i'm not sure i'm going to go ahead and
53:40 : do that right now
53:43 : but i could
53:45 : so
53:47 : oh and i can await it
53:50 : this exact function comes from the
53:53 : child
53:55 : process module
53:58 : trial processes module
54:01 : and also
54:02 : if you want to use the promisified
54:05 : version of this function
54:07 : because normally it isn't there's a uh
54:11 : there's a one liner in the docs that
54:14 : tells you how to get a promisified
54:16 : version
54:17 : of this child process.exec function and
54:20 : this is what i usually use
54:23 : yes
54:25 : yeah
54:26 : is there a reason that you're foregoing
54:28 : the
54:29 : file system
54:31 : removal of directories and files and
54:34 : using oh
54:35 : and switching and using
54:37 : switching to the shell command
54:39 : yeah yeah just because
54:42 : okay i i could use my traverse function
54:44 : to do that too yes
54:47 : but i think it even offers a recursive
54:49 : delete built in
54:50 : does it at this point i'm pretty sure
54:53 : you're on the latest
54:55 : i didn't know that let's see
54:59 : i've always
55:00 : uh installed brimrous but i need to do
55:03 : that
55:05 : oh like uh yeah there's a module that
55:08 : does that for you as well yeah that's
55:10 : true
55:11 : um but i'm pretty sure it's in the
55:13 : native library i i don't know it off the
55:15 : top of my head i think it's like rm
55:18 : or yeah
55:19 : under
55:21 : yeah and they've like moved
55:23 : a little bit
55:24 : but no rm
55:26 : there
55:28 : are removed
55:32 : i know that the one i know is
55:35 : what was it called rmfr
55:38 : something
55:40 : oh rimraf the node module or the module
55:45 : is that how you spell it
55:47 : uh i think uh
55:49 : yes so
55:51 : yeah this one
55:53 : that stands for
55:55 : recursive yeah it's it's same as that
55:58 : but as a node
56:00 : module so you're saying there's this
56:03 : method
56:06 : or yeah you
56:09 : remove dir
56:11 : it can also be used recursively to
56:14 : remove nested directories
56:18 : okay this is the one you're talking
56:20 : about jason
56:22 : i think so i can't remember there's one
56:24 : that they deprecated and one that isn't
56:26 : but there
56:27 : is absolutely a way to do it i think
56:28 : it's that one that's the current one
56:32 : okay
56:34 : so yeah so we could
56:36 : use this
56:37 : is it uh oh i'm gonna assume
56:43 : fs that remove there
56:46 : and it's is it recursive
56:49 : by default or is recursion a flag
56:54 : oh come on it's an option you have to
56:56 : set
57:11 : yeah remove
57:13 : our ender
57:15 : is supposed to not be recursive the
57:17 : recursive option is deprecated so this
57:20 : was added in version 14. yeah
57:24 : and that's where they match the
57:25 : recursion
57:27 : i do not have 14
57:29 : okay
57:31 : so then use remover and use precursor
57:33 : from there
57:35 : okay so i'm going to write it
57:38 : i probably want to do a dry run first
57:42 : i'm going to do it
57:44 : you don't want to remove your operating
57:45 : system
57:47 : removing
57:49 : removing
57:50 : that
57:52 : the dry run
57:56 : okay let's see oh i guess the thing i
57:58 : actually wanted to check was like if
58:01 : uh
58:02 : new this is now so
58:06 : now it's new date
58:08 : so if
58:09 : now minus last modified
58:13 : is greater than two months let's say or
58:15 : whatever you want
58:18 : but i i think two months
58:20 : i have it in my notes as
58:24 : i have in my notes as 1000
58:27 : milliseconds times 60 seconds times 60
58:31 : minutes times 24 hours
58:34 : times
58:35 : 30 days
58:36 : times two that's two months so if it's
58:39 : the difference it's more than two months
58:42 : i'm gonna remove that directory
58:44 : that's the plan
58:46 : let's do the dry run
58:55 : so
58:56 : it's
58:59 : okay it decided to remove this directory
59:04 : which uh
59:05 : was last modified
59:07 : in june
59:09 : which is just over two months
59:12 : and
59:14 : last june of last year
59:16 : and this one is in may last modified in
59:19 : may so we're deleting that and this is
59:21 : from uh
59:22 : last year september of last year so yeah
59:25 : we're
59:26 : we're deleting those
59:28 : uh maybe i should also
59:30 : print out if we're not removing
59:35 : keeping
59:37 : that
59:39 : yeah so if if
59:41 : you get the you get the picture i think
59:43 : if we want to delete it we're gonna do
59:45 : fs dot
59:46 : rm
59:48 : on that sucker
59:52 : uh
59:54 : i can just assume it's done
59:57 : okay
59:59 : our ender
1:00:03 : oh but but recursive is a flag so i have
1:00:06 : to say recursive
1:00:08 : true
1:00:09 : yeah you might have to throw in force
1:00:11 : that's the other thing they call out
1:00:13 : force type of behavior i think that's
1:00:15 : like an os common flag
1:00:20 : okay
1:00:21 : yep uh yeah
1:00:24 : let's just just see if it actually
1:00:26 : doesn't
1:00:31 : oh it's not a dry run anymore even
1:00:34 : though i say it's a dry run so okay it
1:00:36 : should have deleted this folder
1:00:39 : is that true
1:00:40 : is no longer there we do they did it
1:00:46 : yeah we deleted those modules so yeah so
1:00:49 : you can write a little bit of uh
1:00:53 : recursion uh to clean up your disk drive
1:00:56 : um
1:00:58 : and
1:00:59 : yeah i think this this shows that
1:01:02 : this is really really similar to what we
1:01:04 : were doing last week which was
1:01:06 : traversing asts and two things are
1:01:09 : seemingly completely different
1:01:11 : but they're so similar like even down to
1:01:14 : the pattern of traversal that we're
1:01:15 : using and the visitor function and
1:01:18 : everything
1:01:19 : um i think that's
1:01:21 : pretty cool
1:01:22 : um
1:01:24 : yeah any questions at this point
1:01:28 : no okay one more thing i want you guys
1:01:31 : opinion on um
1:01:34 : what topics would you guys like to see
1:01:37 : in the future
1:01:42 : i've been really liking this this little
1:01:44 : streak of
1:01:45 : of doing this type of stuff honestly
1:01:48 : yeah oh i love working with you know
1:01:50 : programming languages and parsing
1:01:53 : all that sort of stuff i never thought
1:01:54 : about using
1:01:55 : the
1:01:56 : this visitor pattern for traversing
1:01:59 : the file system though that's like
1:02:02 : i don't know why i didn't think about
1:02:03 : that but that's uh it's a really good
1:02:04 : idea
1:02:06 : yeah yeah i think one reason might be
1:02:09 : because there are all these tools
1:02:11 : that
1:02:13 : um
1:02:14 : when you google for how to traverse a
1:02:16 : directory there are all these libraries
1:02:19 : that will
1:02:20 : happily do it for you
1:02:22 : and then because you're using their
1:02:23 : abstract action you don't end up doing
1:02:26 : it yourself
1:02:28 : but i think it's kind of fun to do it
1:02:29 : yourself
1:02:30 : and and i mean we're doing the same
1:02:33 : thing we're creating our own abstraction
1:02:36 : as well
1:02:37 : but as you can see it doesn't actually
1:02:39 : take that much code to do at all like we
1:02:41 : wrote this traverse function and it's
1:02:43 : what like
1:02:45 : 15 lines of code
1:02:47 : it just doesn't take very much to do
1:02:49 : this yourself