Let's Talk About Performance

I tell Huiqi about a performance war story that took place last week.

Transcript

The following transcript was automatically generated by an algorithm.

00:01 : all right i did several things this week
00:03 : but i guess the biggest one is i decided
00:06 : to
00:07 : so so last time i said i was gonna try
00:09 : to optimize the
00:11 : the program that reads the log file and
00:16 : generates the pass dates into a history
00:19 : file
00:19 : right um so
00:23 : that program is written in python
00:27 : and um and i said in order to optimize
00:31 : it
00:31 : i i said my hypothesis is that
00:35 : um there's a lot of data duplication
00:40 : whenever i have like a dictionary
00:43 : and and when the the program is
00:46 : update making updates to that dictionary
00:49 : there's data duplication because it's
00:52 : having to
00:53 : re-serialize all the fields of the
00:56 : dictionary
00:57 : yes all the entries of the dictionary
00:59 : every time
01:00 : and then re-saving that and all the
01:03 : values
01:04 : every single time there's an update to
01:06 : that dictionary
01:07 : and my strategy was to not do that
01:09 : anymore and instead
01:11 : have every single field be a pointer to
01:14 : another value
01:16 : so that when a value changes it's only
01:18 : updating that
01:19 : value and then the pointer still doesn't
01:22 : have to change is because it's just
01:24 : pointing to the value that is going to
01:25 : change
01:26 : right um so i
01:30 : i i implemented that idea
01:33 : partially i implemented it enough
01:38 : to be able to uh do that process
01:42 : with some smaller programs like
01:45 : or just subset of the whole python
01:48 : language
01:49 : um i got it to work for uh
01:53 : dictionaries and a little bit with lists
01:56 : but i don't have everything working yet
01:59 : but
02:00 : but um because i'm impatient
02:03 : instead of getting everything to work
02:04 : first before doing the
02:07 : performance comparison i did the
02:09 : comparison
02:10 : you know what the first chance i get
02:15 : so i did some performance comparison and
02:20 : oh kind of want to show you
02:25 : to compare give me one second
02:41 : okay i i so um so first of all
02:45 : um i i i want to talk
02:49 : something about you know performance
02:52 : tuning in general
02:53 : like maybe make it a little bit
02:56 : instructive
02:57 : even though this is like a sort of like
03:00 : a case study
03:01 : where we're talking about this specific
03:04 : instance
03:05 : um i still want to try to
03:08 : you know give some takeaways that i
03:10 : learned
03:11 : while doing this um so one of them
03:15 : is uh sort of using a profiler
03:20 : which is there's pros and cons to doing
03:24 : this approach um but basically i
03:27 : um so if i run
03:32 : um so i have this recreate command which
03:35 : i could run
03:36 : using python 3.
03:39 : uh let's say i do it on
03:43 : so this this dot rewind file that's the
03:46 : log file and then recreate is gonna
03:50 : recreate the path states and uh at the
03:52 : moment
03:54 : it's kind of slow okay i'm gonna
04:03 : i think the fibonacci program
04:06 : is kind of
04:10 : there's the fibonacci program okay i'm
04:11 : gonna make it do a smaller number
04:14 : so this will not take as long um
04:17 : so
04:20 : i was making it a lot
04:23 : to test the performance okay so now
04:26 : it'll complete immediately because it's
04:28 : doing much less
04:29 : stuff maybe i'll make it a little bit
04:31 : more
04:36 : okay okay that took like maybe a second
04:39 : or so
04:40 : um so there's a pro python profiler
04:44 : that you can use which you do it
04:47 : like this uh
04:50 : c profile uh
04:54 : i think you set a sorting option and
04:57 : one of the sorting option is cumulative
05:02 : time cumulative
05:06 : time which stands for cumulative
05:09 : time and it gives you this table
05:13 : and what this table is first of all it
05:16 : gives you the total time
05:18 : okay of the program the total time that
05:21 : the
05:22 : program took and then it gives you
05:25 : for each function that it traced
05:29 : some statistics
05:32 : and this is this graph
05:36 : or this table is sometimes hard to
05:40 : uh make sense of uh there's a lot of
05:42 : data
05:43 : and the the key
05:46 : that you're the thing you're trying to
05:48 : achieve is
05:50 : to um make sense of
05:54 : how how the data relates to your
05:56 : program's performance
05:58 : but basically for each function
06:02 : um well each line is a function
06:05 : and if the function is module that just
06:08 : means
06:09 : the my entire python file might recreate
06:13 : that python file
06:14 : so my entire python file
06:18 : has these characters has these numbers
06:21 : associated with them what are what do
06:23 : the numbers mean uh the numbers
06:26 : this cumulative time means the total
06:29 : time
06:30 : it took between the start of the program
06:33 : and the end of the program
06:36 : this cut time to total time
06:40 : is actually means the cumulative time
06:45 : minus the cumulative time of all of my
06:49 : children
06:50 : function calls okay
06:53 : so so basically i i have my cumulative
06:57 : time and then i would
06:59 : i will take the cumulative time of each
07:02 : of my children
07:04 : add them together and then i'm gonna
07:06 : subtract that
07:07 : away from my cumulative time and that's
07:09 : my
07:10 : total time spent within my
07:13 : function kind of does that make sense
07:16 : sort of it's kind of like
07:17 : the chrome debugger ish
07:20 : yeah chrome debugger uh well it's the
07:24 : pro
07:25 : chrome profiler chrome profiler yeah
07:27 : chrome also has a profiler
07:29 : that has the same kind of the children
07:33 : and it has the children kind of yep
07:36 : uh oh this doesn't have a tree form i
07:39 : believe that one has a tree form
07:41 : which i think is more useful probably
07:45 : but i haven't used the chrome profiler
07:47 : in a while i don't remember
07:48 : what that looks like exactly but anyway
07:51 : uh what
07:52 : i did with this is
07:56 : um like previously
07:59 : i thought that like what i needed to
08:02 : optimize
08:02 : was that data aspect and
08:05 : after studying this a little more i
08:08 : realized that was not true
08:10 : and i and i i i'll show you um
08:13 : what i mean so first of all um
08:16 : in order to read this a little bit
08:19 : easier i actually
08:20 : sort of refactored my code a little bit
08:26 : um oh i'll show this one
08:34 : so first i wanted to know the difference
08:38 : between
08:38 : like i wanted to know uh
08:42 : whether it was parse the parsing code
08:44 : that was slow
08:46 : or the other code that was processing
08:49 : the thing
08:50 : that was slow so i extracted i already
08:53 : had a parsline
08:55 : function but i extracted another
08:58 : function called
08:59 : process command and then that way i have
09:02 : like a clear separation
09:04 : of these two responsibilities
09:07 : i i was assuming that these two were
09:10 : completely uh non-overlapping
09:14 : and independent so it turned out that
09:17 : was not true
09:18 : but it's a good enough approximation
09:22 : so then i can see oh well parsed line
09:26 : took this amount of time whereas
09:29 : uh process command took this amount of
09:32 : time so
09:34 : 0.727 and then if we divide it by the
09:37 : total time
09:39 : which i'm going to take from here 1.439
09:47 : that means the process command is taking
09:50 : about 50 of the time and then the parse
09:53 : line is
09:54 : not much different it's about it's about
09:56 : 50 50 right
09:58 : yeah and and so
10:01 : so i did this kind of analysis and and
10:04 : then if i wanted to break things down
10:06 : further
10:07 : under process command then i
10:10 : take a look at all the functions that
10:13 : process command
10:14 : calls and then sort of do that kind of
10:17 : calculation and uh i sort of did that so
10:21 : like process command
10:22 : calls process call start for example
10:25 : some of the time and it cause process
10:28 : push frames some of the time
10:29 : and then i can sort of tally up these
10:32 : times
10:33 : and and make a build a graph uh which i
10:37 : did
10:38 : for for one of the cases i i made a
10:40 : graph let me see if i can
10:41 : find it yeah this one so i made this
10:47 : i i took the data and put it into this
10:50 : spreadsheet and uh
10:54 : i put in a couple of formulas to total
10:57 : the numbers
10:59 : and i think i made a graph there's the
11:01 : graph
11:02 : it's still loading
11:08 : yeah so that's the graph and uh so so
11:11 : that
11:11 : that's an example of the type of thing
11:13 : that you can do
11:15 : to try to analyze this data but this is
11:18 : saying
11:19 : out of all the things that it's doing
11:23 : inside of process command
11:26 : process command dispatches the control
11:29 : to one of these
11:30 : functions uh over here
11:34 : and then this is the distribution
11:37 : of the total time of those child
11:40 : functions
11:41 : okay and we can see that process
11:44 : call start takes the longest time out of
11:47 : all of them
11:48 : and it takes about 20 of them
11:51 : and so forth
11:55 : so but um later on i
11:58 : realized that process command actually
12:01 : has one
12:02 : part in it that cause parse
12:05 : that calls parse line which makes it
12:08 : slightly overlapping so so these
12:12 : so my numbers actually turned out to be
12:14 : slightly misleading
12:17 : um so that that's the kind of thing you
12:19 : have to watch out for
12:20 : when you when you're doing this and
12:22 : that's i think that's actually a
12:23 : disadvantage of
12:26 : like a table like this where
12:30 : there's it's all it's all tallied
12:33 : based on the function in a flat
12:36 : way instead of a hierarchical tree way
12:40 : yeah so because you you now you could
12:42 : have overlaps
12:43 : like if if this function gets called
12:47 : by like in my case it's like some of the
12:52 : commands
12:52 : like process call start actually calls
12:55 : into parse line
12:58 : the thing i was trying to separate out
13:00 : now
13:01 : i have to do some extra work if i want
13:03 : to really separate them out maybe have a
13:06 : separate
13:06 : function called parse line a versus
13:10 : parse line b and then have this one only
13:13 : call that
13:14 : and then keep them separate yep
13:17 : um but but this this just
13:22 : sort of a hint at like that there might
13:25 : be a better way to visualize this data
13:28 : but like those are things that we can
13:31 : potentially think about
13:32 : later on well i like thinking about
13:35 : using
13:36 : the debugger for for performance
13:38 : profiling
13:39 : stuff um but anyway so so
13:43 : so that i'll give you the sort of the
13:48 : conclusion that i came that i came to
13:51 : um after looking at
13:55 : the data in a more
13:58 : more rigorous way than i had before so
14:01 : basically i had
14:02 : a run where i ran this
14:05 : profile or ran my new version
14:09 : of recreate which i called i called that
14:12 : recreate too
14:14 : okay i ran that and i compared the
14:17 : result
14:18 : to the recreate one
14:21 : um and
14:25 : and i ran it on a relatively large log
14:28 : file
14:29 : um and i realized that
14:34 : first of all the total performance
14:37 : um like i did it with the fibonacci
14:40 : i'll call it fib and i ran it for
14:46 : when it tried to calculate the fibonacci
14:49 : number at index 20. okay
14:52 : and then it was like recreate that pi
14:56 : took 9.14 seconds
14:59 : recreate to that pi took
15:03 : 8.099 seconds so
15:06 : it was slightly faster but not fast
15:09 : enough that
15:10 : it's definitively faster
15:14 : and then and then under there i
15:17 : found that parse line
15:20 : the parsing parseline is usually the
15:23 : it is the entry point for the parser by
15:26 : parsing
15:27 : each line of the log file and then
15:30 : process command
15:32 : process command is after you parse the
15:34 : line now you're going to process the
15:36 : thing
15:37 : do stuff with data structures and save
15:39 : it into the database
15:44 : so in the in the old one
15:48 : it was you know those numbers and in the
15:51 : new one
15:59 : and the new one is
16:03 : in the new one it was these numbers so
16:07 : um
16:08 : so you can see the it's it's like 50 50
16:12 : where
16:13 : about 50 50 but the parsing is actually
16:16 : slightly
16:17 : worse than the processing in both cases
16:21 : um
16:23 : in in the new one the processing
16:26 : is slightly better than the
16:29 : than the old one and that could be
16:31 : attributed to
16:33 : the the new scheme that i'm using
16:36 : right um but but but it wasn't a huge
16:39 : difference and and also
16:43 : um one of the one of the things that
16:51 : this execute is like the database
16:54 : call the total of the time
16:58 : that's spent like interacting with the
17:01 : database
17:01 : the sql api sql lite api yes that amount
17:06 : uh and the old one uh
17:10 : whereas in the new one
17:16 : it was that
17:19 : uh there's actually it's even slightly
17:22 : worse
17:22 : than the old one so there isn't like a
17:26 : there isn't a clear improvement
17:29 : in terms of the amount of time dealing
17:32 : with the database however
17:34 : when it came to serializing
17:38 : the old one had some time dedicated
17:41 : to serializing the data before the data
17:44 : goes into the database
17:46 : right whereas the in the new version
17:49 : there was no serialization needed so the
17:52 : new version has
17:53 : zero serialization overhead
17:58 : um but but but other
18:02 : like you can see that the other part is
18:05 : actually
18:07 : 2.6 like if i subtract
18:11 : that number from that number we get that
18:13 : number
18:14 : [Music]
18:17 : and then over here if i subtract these
18:21 : two numbers from this number
18:23 : i get that number which which shows that
18:28 : this other other than the stuff dealing
18:31 : with this
18:32 : inserting data into the database there's
18:35 : still
18:36 : a significant amount of work outside of
18:38 : that
18:39 : yeah so my scheme
18:42 : that my new data scheme was targeted at
18:46 : specifically these two things
18:50 : however that's not necessarily a huge
18:53 : portion
18:54 : of the overall process yeah that makes
18:58 : in a way i have eliminated this
18:59 : serialized part
19:02 : but that serialized part is
19:05 : if i divide it by the entire thing which
19:08 : is nine
19:08 : seconds that's only 11
19:12 : of the big picture and and yeah maybe i
19:16 : have done something
19:17 : but it's clearly not a huge amount
19:21 : that's the overall performance um so
19:25 : um so another thing is i i
19:28 : made a i made a
19:32 : uh a code example where it
19:35 : that clearly favors my new approach
19:40 : which is it creates a enormous
19:43 : dictionary
19:44 : it's like of like 10 000 entries
19:47 : in the dictionary so that that's gonna
19:50 : clearly
19:50 : favor mine and when i did that
19:54 : what i found was that
19:58 : let's see i'll call that the big dick
20:02 : big dictionary um
20:06 : when i did that yeah that the with with
20:09 : a big
20:10 : big dictionary example
20:15 : the recreate the old one
20:18 : took seven 7.071
20:21 : seconds and the new one
20:28 : took 0.513 seconds so there's a huge
20:32 : difference
20:33 : if i have a enormous dictionary in the
20:35 : program
20:36 : that mine is much much better
20:40 : but but if we can drill down into
20:43 : the specifics to see where
20:46 : that all that time is being spent
20:49 : so in process command
20:52 : it was 6.570 so most of the time was
20:57 : being spent in process command
20:59 : in this case now whereas the parse line
21:03 : only took 0.286
21:07 : seconds in this one so when when it's
21:10 : really bad
21:11 : for the old
21:15 : old data scheme it was because the
21:17 : process command part
21:18 : was getting really bad and more
21:20 : specifically it was the serialization
21:23 : part
21:24 : that was getting really bad most of the
21:26 : time was spent in
21:28 : the serialization
21:32 : and and this is actually 89 of the total
21:35 : so so what i'm saying is if
21:39 : if there's a situation like that
21:42 : clearly favors the new scheme that i
21:46 : worked on
21:47 : then that will be because
21:51 : the uh the old version
21:54 : of the code is doing way too much
21:57 : serialization
22:01 : but but then i also
22:04 : had to ask like
22:08 : in the cases in the practical cases that
22:11 : i have encountered
22:13 : like because when i'm trying to optimize
22:16 : i'm trying to optimize for my
22:18 : own workflow and like actual scenarios
22:21 : that i encounter that are slow
22:23 : so i chose some of those um
22:26 : that i encountered that are slow one of
22:29 : them was the fibonacci one but another
22:31 : one is i try to
22:32 : have the recreate script generate
22:36 : uh the pass state for itself
22:40 : so i had recreate create the log for
22:43 : recreate i read the recreate log
22:47 : back in okay i did that
22:51 : i found that more or less still the
22:53 : parsing
22:54 : was about 50 50 with the process command
22:58 : which shows me it it's not a case of
23:01 : it's doing way too much serialization
23:03 : in that case and in the practical case
23:06 : even though question array okay
23:09 : that's really bad for it it practically
23:12 : i haven't
23:13 : uh hit those cases yet
23:19 : so that one was that one was like
23:23 : when when i have recreate
23:29 : running the recreate rewind file i got
23:32 : the total time to be
23:34 : 33 seconds and the parse
23:38 : line time was 15 seconds
23:41 : and the process command time
23:45 : was 17 229 seconds
23:48 : uh i didn't do this one with the
23:52 : new version of recreate because it's not
23:55 : there yet in terms of features to be
23:57 : able to do this one
24:00 : okay um but
24:04 : okay but oh and and then if we break it
24:07 : down a little bit more
24:08 : inside of process command the db execute
24:12 : um took 6.222 seconds and then serialize
24:19 : took three point seven five two seconds
24:22 : and then other than that it took
24:26 : soon two five five seconds so
24:29 : curly like at the serialize is
24:32 : there's some amount of serialization but
24:34 : it's still not a
24:36 : huge portion of the overall picture
24:40 : like the other is even greater than
24:42 : serialized
24:45 : and then so when i asked the question
24:47 : like what what is all this stuff
24:49 : in other what what are you doing other
24:51 : than
24:53 : inserting data into the database and
24:55 : serializing the string
24:57 : the only answer i could come up with was
25:00 : just that
25:00 : it's just doing the like
25:05 : doing stuff with dictionaries doing
25:07 : stuff with lists
25:10 : if statements like all the normal things
25:13 : you do with the
25:14 : with the programming language right like
25:17 : this
25:18 : these are just like core features it's
25:21 : kind of of python
25:23 : so that's that's the point where i said
25:25 : okay it's time to rewrite this in
25:28 : c because okay
25:31 : because because it's so because if i
25:34 : look at this
25:35 : big picture it's like this time
25:40 : because my initial theory was hey it's
25:43 : it's the database that's slow it's
25:46 : usually the database that's slow
25:48 : when you're optimizing things usually
25:50 : your this database is the slowest thing
25:53 : so you have to optimize the database
25:55 : but in this case it's actually not
25:57 : because if you take
25:58 : that number okay there's a little bit
26:01 : more
26:02 : because other than the database execute
26:04 : there's also a commit
26:05 : call that
26:08 : that's outside of outside of parsline
26:13 : that is taking that amount of time so
26:16 : but if i but that's insignificant
26:20 : so if i take that and divide it by the
26:22 : total number
26:26 : it's 20 of the overall
26:29 : time is being spent dealing with the
26:32 : database
26:33 : it's only 20 the other 80
26:36 : is not database at all that's just
26:40 : python okay so basically i decided to
26:44 : rewrite this
26:45 : thing in c to get maximum performance
26:49 : out of it
26:50 : so y c is much faster than python in
26:53 : this case
26:54 : yeah that's a good question um why is c
26:58 : much faster than python so
27:01 : um python is an interpreted language and
27:04 : that actually have gotten to take a peek
27:07 : at
27:07 : what the interpreter does um but
27:10 : essentially it's almost like
27:15 : um
27:15 : [Music]
27:19 : there's an extra level of like like you
27:21 : you have cpu has instructions
27:24 : cpu has instructions that looks like
27:26 : like
27:28 : you know you you put your code in here
27:31 : and each line it's some bits
27:34 : that represent an instruction telling
27:37 : the cpu what to do
27:38 : right um python has a virtual machine
27:43 : so these are instructions
27:47 : but python has a virtual machine
27:50 : we call it the python interpreter
27:55 : python interpreter is acting like a cpu
27:59 : and it's reading in line by line and
28:02 : then it has to
28:03 : look at the bits to determine what
28:07 : instruction this is whether this is an
28:09 : add
28:10 : or load an object or initialize an
28:13 : object or call a method or whatever
28:16 : and then it'll do the work okay
28:19 : um there's a so each of this
28:24 : each of these things these actions that
28:26 : it does like
28:27 : hey what is this instruction oh it wants
28:30 : me to make a method call now i need to
28:34 : fetch
28:35 : the object that contains the method now
28:38 : i need to load the method
28:39 : based on the method name i i need to
28:43 : based on the method i like to look up
28:45 : what the method object is oh now i have
28:47 : the
28:47 : method object i'm going to put the
28:48 : method object onto the stack
28:50 : and then so that there's a separate
28:52 : instruction after that to actually call
28:54 : the method
28:54 : and when i actually call the method i
28:56 : have to take the arguments off of the
28:59 : stack
28:59 : and pass it into the method object in
29:02 : order to call it
29:03 : so there's actually a whole bunch of
29:06 : sort of infrastructure
29:08 : level code that's going on
29:12 : that's that's just making the language
29:14 : itself
29:15 : work it's just a lot of overhead it's
29:18 : like you have to get through all the
29:20 : machinery
29:22 : it takes to just to do a method call
29:26 : before the stuff that the method does
29:28 : actually
29:29 : even happens and that's actually a
29:31 : substantial amount of overhead
29:35 : whereas if you compile it in c
29:38 : the code in c just gets compiled down to
29:41 : the instruction sets
29:43 : that are fed directly into the cpu and
29:46 : there's no
29:47 : infrastructure overhead involved at all
29:50 : it's just doing what your program is
29:52 : trying to do
29:54 : gotcha so that that that's a short
29:58 : explanation um
30:02 : and but basically um what i did
30:05 : so far which is very promising is
30:09 : i wrote just the parse line part
30:13 : i i did an experiment where i wrote the
30:16 : parser in c
30:18 : okay and then i i
30:22 : tested that head to head against python
30:25 : so it was like c doing just the parsing
30:28 : versus python doing just the parsing
30:31 : okay
30:31 : and the end result was like the c c
30:34 : was uh 34 times faster
30:38 : oh wow that's crazy
30:42 : um so uh and
30:45 : that is like very very like a lot
30:48 : faster there's multiple reasons to why
30:51 : it's
30:52 : like i in python i was also using the
30:55 : eval
30:56 : function which was slow and
30:59 : there were there are ways i can do
31:01 : without calling the eval function
31:04 : to make it much faster but other than
31:07 : that it was just
31:08 : python itself is slow okay but yeah
31:11 : by by rewriting just the parsing and c
31:14 : i got it to be much much faster and when
31:17 : i
31:18 : did some rough calculation like if i can
31:21 : make this number
31:23 : like 34 faster and if i
31:27 : if i assume that i assume the db
31:30 : execution part
31:31 : that's the part i cannot improve that
31:34 : much
31:34 : because it's calling into the sqlite api
31:38 : and the performance of that is
31:40 : determined by what the sqlite api is
31:43 : doing
31:43 : right so i'm to assume this stays
31:47 : fixed but i'm going to assume the parts
31:50 : that is python code executing i'm going
31:53 : to assume i can get it to maybe
31:56 : around that number i'm going to i'm just
31:59 : gonna call it like
32:00 : one order of magnitude faster
32:04 : okay have you heard of that this term
32:08 : before
32:08 : no lag need
32:12 : to one order of magnitude faster
32:16 : um and and which usually means
32:19 : it's 10 times faster it's like
32:22 : it's one power faster it's like
32:25 : whatever the number is x of n we just
32:28 : we just like decrease this n
32:32 : by one okay that's pretty 34 is
32:35 : around there i i decrease this exponent
32:39 : by
32:39 : n that's how much faster i made it
32:43 : yeah and you can also uh like
32:46 : make this other part uh in the process
32:49 : command faster right exactly
32:53 : that's what i'm banking on right i mean
32:55 : like first of all the serialized
32:57 : part i can probably exit out if i use
33:00 : the new approach
33:02 : there's no more serialize and then but
33:04 : this other part
33:06 : i'm betting that i can do one order of
33:08 : magnitude faster
33:10 : for this number as well yeah so so if
33:13 : that did happen
33:14 : it would end up be like well as i
33:17 : calculated before
33:20 : as i calculated i i think the
33:23 : database oops i think the total database
33:28 : for this
33:29 : particular scenario the total database
33:31 : time is seven
33:32 : seconds and then
33:35 : that means the parts that's not database
33:38 : time
33:39 : and it's python is 26 seconds
33:42 : what if i can what if i all of that can
33:45 : be 34 times
33:47 : faster that means all of that will
33:50 : now be 0.7 seconds
33:54 : and then so if if if i can do that
33:58 : i add that back to my database time of
34:00 : 10 7
34:01 : seconds the overall time will be reduced
34:04 : from 33 seconds
34:05 : to around eight seconds and that's a
34:08 : huge improvement
34:09 : right yeah that's significant yeah yeah
34:12 : sure
34:12 : so that's what i'm banking on uh but but
34:15 : there's a bunch of work to do
34:17 : like to to port everything over from
34:20 : python
34:20 : to be rewritten in c so i'm working on
34:22 : that right now
34:26 : well that's really cool yeah i think
34:29 : that's a great
34:31 : discovery for sure yeah i i guess
34:34 : like if i were to try to summarize what
34:37 : i learned
34:38 : in this in this story
34:42 : it would be um
34:48 : one one of them is one of them is just
34:51 : like look for
34:52 : look for improvements by
34:57 : one order of magnitude like look
35:00 : you know like look for um
35:04 : look for 10 times
35:08 : improvements or one order of magnitude
35:15 : like don't don't don't try so hard
35:18 : don't try so hard to work on things
35:21 : that'll give you
35:22 : like 50 improvement
35:25 : those are not as meaningful you you you
35:28 : really want to find something that
35:30 : gives you 10 times better or 50 times
35:33 : better or something like that
35:36 : yeah and you found a bottleneck well
35:39 : it but i i guess in this case like there
35:42 : is not like just
35:43 : one bottle like a bottleneck that's very
35:46 : obvious
35:47 : yeah it's actually just python is slow
35:50 : and but i had to
35:52 : do some deductive reasoning
35:55 : like well first of all i had to
35:58 : visualize
35:59 : the results of the profiler which
36:02 : was not easy like i didn't get it the
36:05 : first time
36:06 : i looked at that data but i tried to you
36:09 : know
36:09 : i had to look at that data in a
36:12 : different way
36:12 : before i came to that conclusion
36:17 : but yeah but i i concluded that it was
36:20 : the python that was slow
36:23 : uh yeah but you need to like you really
36:27 : really want to understand what's going
36:29 : on like
36:31 : the more you understand the more
36:34 : you understand what all the code is
36:38 : doing the better
36:42 : and because oh and another thing i would
36:44 : say it's like
36:45 : performance getting code to run fast
36:49 : is not like
36:53 : it's not like running fast it's it's not
36:55 : like
36:57 : usain bolt running super fast
37:00 : because you know he's just more powerful
37:04 : runner
37:04 : than ever it's not about power making
37:07 : code
37:07 : run fast is more about
37:11 : being extremely stingy with your
37:14 : resources
37:16 : if that makes sense it's like it's like
37:19 : the accountant that tries to cut
37:22 : expenses
37:23 : in all possible places that that's what
37:26 : it's
37:27 : more like okay like like so you have to
37:30 : understand
37:31 : like it's like cost cutting it's like
37:34 : like it's more like like cost
37:38 : cutting accountant
37:42 : not not usain bolt
37:46 : like you're trying to figure out oh
37:49 : there's all
37:50 : this code that was executed
37:53 : which parts can i cut out
37:56 : you look like you look at like oh
37:58 : there's a lot of code
38:00 : that's spent doing this thing
38:03 : can i cut that out um
38:06 : and and the only way you can know what
38:09 : you can cut out and what you cannot cut
38:11 : out
38:12 : is by understanding the code more and
38:15 : then
38:15 : also look for you know do the
38:18 : do the pieces that has the most value
38:22 : right like you you don't you don't you
38:25 : don't spend a lot of time
38:26 : cutting out the small stuff you want to
38:30 : spend time
38:30 : cutting out the really big stuff to make
38:32 : a huge impact
38:34 : yeah that makes sense