Real World Leaky Abstractions 2: Sharing TypeScript

In this video I share with Huiqi a war story where I try to share TypeScript definitions among 3 projects: a nest.js project, a React-create-app project, and a React Native/Expo project. It is a classic case of a leaky abstraction: where something which seems like it should be straightforward ends up sendings me through a long series of rabbit holes.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : uh i'll do one more example
00:03 : the example is something i've been
00:06 : dealing with uh
00:07 : last few weeks we have an app
00:10 : which has a back-end
00:13 : a mobile app and a web app
00:16 : and then and all so these are like three
00:20 : sub
00:20 : projects within the main project and all
00:23 : of them
00:24 : are developed in typescript
00:27 : the backend is developed in nest gs
00:31 : the the mobile app is developed in uh
00:34 : react native slash expo and
00:38 : the web app is developed in react okay
00:42 : so there's a desire in some cases to
00:45 : share
00:46 : typescript definitions among the three
00:49 : projects
00:51 : and developers so far have been doing
00:54 : this
00:56 : dot dot dot up to the parent project
00:58 : level and then
01:00 : going back in and usually they pull
01:03 : interfaces out of the back end project
01:06 : right like maybe the mobile app will
01:09 : pull in some interfaces that were
01:11 : defined in the backend project
01:12 : and stuff like that because the data
01:15 : shape is there
01:16 : right and they don't want to duplicate
01:18 : the code
01:19 : um but uh the only works for the type
01:22 : definitions that
01:24 : don't require any javascript to be
01:26 : emitted
01:28 : so so notably it doesn't work for enums
01:32 : because edoms require javascript to be
01:36 : emitted
01:37 : okay in order to work we just got stuck
01:40 : we don't know how to do this
01:41 : and then we did research about how to do
01:44 : this and
01:46 : and i actually tried multiple ways i i
01:50 : i did like experimental approaches
01:52 : multiple ones maybe four or five
01:55 : basically the problem is the
01:58 : concerning typescript right there's a
02:00 : typescript compiler
02:01 : but the typescript compiler is embedded
02:04 : inside of
02:04 : another framework but we have three
02:07 : frameworks not just one we have an sjs
02:10 : framework in one case in another case we
02:14 : have a
02:15 : react native slash expo framework and
02:18 : another case we have a
02:19 : create react app framework each each one
02:23 : handles typescript differently okay
02:26 : each one has a different way of handling
02:30 : typescript compilation and and the tools
02:33 : are
02:35 : you know embedded within the larger
02:37 : framework so so i actually dug
02:39 : into each one they looked at
02:42 : how they wired up things internally
02:46 : more or less and i uncover interesting
02:49 : things so so
02:51 : next js for example it just uses the
02:53 : typescript compiler
02:55 : okay directly and that was the easiest
02:58 : one to configure
02:59 : to do what i wanted it to do
03:03 : the react create app one the the
03:06 : web-based app
03:08 : um that internally it uses webpack
03:12 : okay react has this notion of eject
03:16 : right okay did have you heard about the
03:19 : eject
03:20 : okay all right we might have to do
03:23 : another
03:24 : episode just about eject because i have
03:26 : a lot of thought about the concept of
03:28 : eject react create app has webpack
03:32 : pre-configured in a very specific way
03:35 : okay and and there's no they don't give
03:39 : you a way to
03:40 : change that configuration
03:43 : okay however they have i i believe
03:46 : this is how it happened they probably
03:48 : got complaints
03:50 : from the community that like but i
03:52 : really need to customize this thing
03:54 : why are you not letting me do that so
03:57 : they i think
03:58 : then they added this feature called
04:00 : eject
04:02 : so if you eject if you press the eject
04:05 : button
04:06 : oh yeah like you the way you do it you
04:08 : npm run eject
04:10 : and then you eject when you eject you're
04:12 : basically
04:13 : no longer using the
04:16 : v8 react create app framework
04:19 : after you reject but what what it will
04:22 : end up doing is spit out a whole bunch
04:24 : of
04:24 : webpack configuration and other
04:26 : configuration file
04:27 : and also scripts that you would run
04:31 : instead of running it through their
04:33 : tools so they will spit out like
04:35 : three node.js scripts right the start
04:38 : script the build script and then the
04:40 : test script
04:41 : and then you would run those node.js
04:43 : scripts instead and
04:44 : and you're not dependent on react create
04:48 : app so that's sort of the
04:50 : compromise that they reached with the
04:53 : with the people
04:54 : that really really want to customize
04:56 : things i have more
04:58 : thoughts about the concept of ejects but
05:00 : uh i'm gonna skip it for now
05:02 : but well the community then figured out
05:05 : that we added on tools to allow you to
05:08 : customize
05:09 : certain things without needing to eject
05:12 : that's what angular does we have to do
05:15 : this
05:15 : to our regular project which is we need
05:18 : to
05:19 : uh slightly modify the webpack
05:21 : configuration so there are so
05:23 : in angular there's the angular.json and
05:26 : there are uh configurations that you can
05:28 : just add your own
05:30 : uh webpack uh configuration but it's an
05:33 : add-on it's not like you can just do
05:36 : everything yourself yeah
05:37 : it's pretty hilarious that in order to
05:40 : customize it you have to add an
05:42 : add-on first um but anyway
05:45 : um oh i i tried a symbolic approach
05:48 : i i thought hey i could just create a
05:51 : sim link
05:52 : and then have each sub project
05:56 : have a sim link directory in link
05:59 : oh what is sim link uh symlink is a
06:02 : symbolic link it's like a pointer
06:04 : it's a directory pointer on your file
06:06 : system you can
06:07 : create an imaginary directory that in
06:09 : reality is a pointer to another
06:11 : directory okay so my thought is like
06:14 : each sub project
06:15 : would have a subdirectory which is an
06:18 : imaginary pointer to an
06:20 : outer directory that that is
06:23 : like the shared folder of
06:26 : source code that's shared between the
06:28 : three projects
06:29 : and at first i thought it worked but
06:32 : then it only worked for
06:34 : one of those three setups and then for
06:37 : the two other setups i had to do extra
06:39 : work to either get it to work like for
06:42 : the
06:42 : react native setup react native it turns
06:45 : out react native
06:47 : doesn't use webpack and instead
06:50 : facebook decided to create their own
06:52 : bundler specifically
06:54 : called metro specifically for native
06:57 : apps
06:57 : so i dug into the metro and it turns out
07:00 : metro uses a file crawler from
07:02 : jest that the test framework
07:05 : and as i dug into those things um
07:08 : i was able to figure out how to hack
07:10 : their source code to get sim links to
07:12 : work
07:13 : but even even after that i decided
07:16 : that's not worth it
07:17 : because like what are the chances i'm
07:19 : gonna get them to take my changes
07:22 : for the metro case basically i
07:25 : used a custom
07:28 : plugin the metro allows you to configure
07:31 : it now metro is better about
07:33 : configuration
07:34 : than the react stuff so there's a way to
07:37 : plug in a module resolver
07:39 : and then so when it's trying to module
07:41 : this module i just redirect it to my
07:44 : common library directory i can do that
07:47 : so basically
07:48 : that was like a really really long
07:51 : journey
07:52 : it was a little bit extreme but i guess
07:56 : the summary is always all we were trying
07:59 : to do
08:00 : was reuse some typescript definitions
08:03 : between three projects
08:06 : and uh it's actually very very
08:08 : non-trivial
08:09 : um now there's a mono repo
08:14 : enabler project called nx it's a
08:17 : okay they're calling themselves smart
08:19 : extensible build framework
08:21 : like we could have bootstrapped our
08:24 : project
08:24 : using nx and then
08:28 : basically all the other digging that i
08:31 : had to
08:31 : go through the people that made
08:35 : nx they did that and more
08:38 : they did all of the pain that i was
08:40 : going through
08:42 : and they they figure out how to
08:44 : integrate with each
08:45 : main framework that's in the community
08:48 : and so if you want a
08:51 : mono repo is what they call it like a
08:53 : larger project that can house
08:55 : multiple smaller projects like this
08:58 : and it's in the javascript ecosystem
09:03 : uh they will do that wiring up for you
09:06 : like like the wiring up that i did with
09:08 : metro for example i i stole the
09:11 : technique from
09:12 : nx because those people have figured it
09:14 : out uh before me
09:16 : um so is this abstraction too
09:20 : yes and that's what that's kind of the
09:22 : thing
09:23 : uh yeah it's it's like it's it's really
09:26 : funny actually
09:27 : if you think about it because the
09:30 : problem
09:31 : is the problem that we faced is that
09:34 : these frameworks have leaky abstractions
09:39 : and it was painful to sort of customize
09:43 : those things because i i had to
09:47 : dig into the internals of each one of
09:49 : the big frameworks
09:51 : and the solution that nx provides
09:54 : is hey don't worry about it
09:58 : you don't have to do that because i will
10:01 : make that all
10:02 : easy for you and guess what what happens
10:05 : if nx doesn't work like for whatever
10:08 : reason
10:08 : what if there's a leaky abstraction in
10:11 : annex which i'm sure there will be
10:13 : then then you know okay
10:17 : now what do i do not only will i have to
10:20 : understand how what nx is doing how nx
10:24 : works but i will also have to understand
10:26 : the
10:27 : the framework that annex is trying to
10:29 : abstract away from you so that you don't
10:31 : have to know the internal i have to know
10:33 : even more or most people at this point
10:36 : they would just give up
10:37 : they will they will do something else
10:41 : they'll probably like say you know what
10:44 : let's use go
10:48 : just rust we're not gonna do this
10:50 : anymore