Visual Concepts: Social Graphs and Wikipedia Edit Histories

This is the second episode where I share with Huiqi (and you!) inspiring visualization concepts from the book: Beautiful Visualization. First we talk about Valdis Krebs' social graphs. Then we talk about Martin Wattenberg and Fernanda Viegas' visualization of the edit history of Wikipedia articles: http://hint.fm/projects/historyflow/.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : so all right so this is a
00:04 : this is like a network um the but the
00:07 : the starting data
00:10 : like the the this is like a famous data
00:13 : set
00:13 : that's used in a lot of papers
00:16 : okay and this data set is
00:20 : basically saying hey there's these
00:24 : 15 or so women and then
00:27 : these women go to these
00:31 : group these events or these classes
00:34 : or something okay so
00:38 : and and we have data about which
00:41 : events these women go to every week
00:46 : and then therefore we can tell the
00:50 : social graph of this group of women
00:54 : based on which of the women go to the
00:57 : same event
00:58 : basically okay
01:01 : based on that so so so the way they
01:05 : did it there's multiple ways you can
01:08 : sort of color this information but
01:12 : the way that this author did it and let
01:15 : me read this author's name
01:17 : [Applause]
01:19 : valdes craps that's the author
01:23 : and he's saying
01:27 : um a woman x is connected to
01:31 : women why if they both
01:34 : attended a certain event
01:39 : um but they they can give different
01:42 : strengths
01:43 : to that connection so for example if
01:46 : women
01:46 : x and women y they both have
01:50 : two events in common then they have a
01:53 : stronger strength
01:54 : than if they only have one even in
01:56 : common okay
01:57 : that makes sense so so what he did is
02:01 : like he starts
02:02 : with like a really
02:06 : big strength he starts with strength of
02:10 : four
02:11 : the strength of four graph so that so
02:14 : he's only gonna show the women
02:16 : that are connected based on that
02:21 : they um they go to four of the same
02:24 : events together
02:25 : so if they did that um
02:29 : you get this uh small graph here
02:34 : that shows there's four women who
02:37 : go to four same events together
02:41 : okay there are four events where if you
02:44 : go there you will see all four of these
02:46 : women every time
02:48 : okay and then there's a separate group
02:49 : of four women
02:51 : there uh i mean three women that go to
02:54 : a distinct like uh
02:58 : some other four events but are not
03:00 : connected with these
03:02 : four right yes
03:11 : three other women they're very strongly
03:13 : connected together based on what events
03:16 : they go to
03:17 : right so that's this is the this one
03:20 : over here is the strength
03:21 : four graph and then if we reduce it down
03:24 : to the
03:24 : if you allow this less strength like
03:27 : strength
03:28 : three you'll get more women showing up
03:31 : in the graph
03:32 : like this okay but you can in this data
03:35 : set you can still
03:36 : see there's two distinct groups of women
03:39 : that are really really connected
03:41 : together but there's no overlap
03:43 : between the groups and then if you add
03:46 : even more at allow the weaker strengthen
03:50 : oh it's not this one sorry
03:54 : uh if you then allow more weaker
03:58 : strength and you get
03:59 : at the end you get this graph if you if
04:02 : you
04:02 : allow strings one you can see there's
04:06 : two really closely connected groups
04:11 : but there's some other people that are
04:13 : sort of not as
04:14 : connected to the middle of either of the
04:17 : graphs
04:20 : so um
04:24 : i think that could be potentially very
04:27 : interesting
04:28 : for for applying that
04:32 : concept to uh get repositories
04:36 : uh if you instead of using the
04:39 : event you use a file
04:42 : in the repository like two people are
04:45 : connected
04:46 : if they both edited this file
04:49 : in the repository and and you use that
04:52 : to build up a social graph
04:55 : um you might get some interesting
04:57 : information out of that
05:00 : [Music]
05:02 : interesting well do you think um like
05:05 : the social
05:06 : network uh software use this kind of
05:09 : stuff to calculate
05:10 : like how people should be connected
05:13 : they probably do yeah they have so much
05:16 : information
05:17 : i'm sure they've looked at this yeah
05:20 : like if you're facebook and you're
05:22 : working
05:23 : you're working on the facebook team
05:27 : that's doing data analytics i'm sure
05:30 : they
05:31 : they all know this kind of research
05:35 : um so i think that could potentially be
05:37 : interesting for um
05:39 : for visualizing
05:42 : github history but there could be other
05:45 : ideas
05:46 : that are that you know this
05:49 : is applicable for um you think
05:53 : this can be used to see whether the
05:55 : certain modules are coupled
05:57 : to each other oh yeah yeah yeah yes um
06:01 : that's another thing is because there's
06:04 : also a
06:06 : visualization not for the women but for
06:09 : the events
06:10 : okay so this is a visualization for the
06:13 : event
06:14 : and it kind of tells you um how do i
06:18 : explain this
06:19 : the the events that have these stronger
06:22 : lines
06:23 : connecting them are the ones that are
06:28 : kind of have a lot of strongly connected
06:31 : people
06:32 : to them basically um
06:36 : so so maybe you can think of them so
06:39 : like
06:39 : event a is connected to even b
06:44 : if they were both attended by the same
06:47 : woman
06:48 : see so okay it's like
06:51 : a two like the line
06:56 : if if there's a line between two events
07:00 : the line gets stronger the more
07:04 : people there are there are that
07:08 : go to both of the events okay
07:12 : strong line means there's more
07:15 : women let's say 10. there's 10 women
07:17 : that go to both of these
07:19 : two events that makes sense um
07:22 : so i guess what you can say about that
07:26 : is like and they kind of strategically
07:28 : arrange these
07:30 : nodes so that the strong
07:34 : links are sort of you know laid out in
07:37 : the middle
07:38 : that is true yes um and and you know i
07:41 : don't know if
07:42 : a person did that or an algorithm did
07:44 : that but
07:45 : they just arranged it that way so you
07:47 : can sort of see oh there's
07:49 : this you know the around the center
07:52 : is where the important events are so
07:56 : based on that you can
07:58 : think about oh if i want to
08:02 : be influential if i want to influence
08:05 : the women
08:06 : in some way with with like a message
08:10 : like or i want to sell a product to them
08:14 : i guess usually it's like that oh i have
08:16 : this
08:17 : social network app uh-huh these women to
08:20 : use or something but it could
08:22 : be any kind of thing or maybe it's like
08:24 : a political message you want them to
08:26 : vote for
08:28 : a certain political candidate whatever
08:30 : it is
08:31 : then you'll know to well
08:34 : i want to infiltrate them by going
08:37 : through the strong links in the graph
08:40 : gotcha um yeah so so that's interesting
08:45 : um so maybe maybe like you could do the
08:49 : the same thing with the code repository
08:53 : where like you can see where the strong
08:55 : links are
08:56 : the strong links are surrounding those
08:59 : some set of files that get
09:03 : sort of heavily changed i think
09:06 : so you can kind of see the set of files
09:09 : that are
09:10 : important at least in this way of
09:13 : looking at things
09:16 : that makes sense i think for
09:19 : for new people going into a code base
09:22 : this can be useful um to to just
09:27 : know what files are more worthwhile
09:32 : to look for them maybe
09:35 : yeah or or what people to ask
09:38 : if they have questions um
09:43 : yeah okay and the last one i want to
09:44 : show is
09:46 : there's a couple of guys martin
09:48 : wattenberg
09:52 : so they came up with a way of
09:54 : visualizing articles
09:57 : in wikipedia so this is a visualization
10:00 : of the article
10:03 : on wikipedia um and
10:08 : you can see at the top you can see these
10:11 : tick marks
10:12 : here or you can also see these straight
10:15 : lines in the background
10:17 : so each each straight line is one
10:20 : version
10:21 : of the art okay so somebody has
10:24 : added it between this straight line and
10:26 : this straight line
10:28 : the horizontal line or the diagonal
10:30 : lines
10:31 : in some cases is is telling you
10:35 : um for each word i believe
10:38 : in the article it's saying
10:42 : this this diagonal says this word
10:46 : from the previous version moved up
10:50 : to this location in the article in the
10:52 : next version
10:53 : okay so so like it's that word
10:56 : survive the edit
11:00 : when there's a diagonal it move the word
11:02 : move from here
11:04 : and then in the next version it moved
11:05 : down here and then in the next version
11:07 : it
11:08 : moved down here and so forth and and the
11:11 : color is
11:12 : based on the author of that word
11:16 : the the person who so so at the
11:19 : beginning
11:20 : it was just one author
11:23 : this light pink person
11:27 : and then the light and then the in the
11:29 : next edit is still mostly
11:30 : light pink but there's this a little bit
11:32 : of green up here
11:34 : so there's a second author that wrote a
11:37 : little bit on top
11:38 : but you know some words got shifted
11:42 : up that means some of the words over
11:45 : here probably got deleted
11:47 : and maybe the green author decided to
11:51 : rewrite the beginning chunk
11:54 : and then delete a bunch of or just
11:56 : shortened it
11:58 : i like this i you can see the movement
12:01 : of the content
12:02 : uh over time um and if you're wondering
12:06 : what this
12:06 : zigzag stuff is that is
12:10 : yeah that is a edit war where
12:15 : so clearly there's this content
12:19 : is stays the same starting from here but
12:22 : it just moved up and down
12:24 : and that's because there's a not a
12:27 : section
12:27 : here somebody wanted to add this section
12:31 : but then another person decided to
12:33 : delete it immediately
12:35 : and then the first person want to add it
12:37 : again and then they wanted to delete it
12:38 : again
12:40 : that is funny i think github should have
12:42 : this too
12:43 : so that i can see who messed up the code
12:47 : yeah so this is like a typical
12:50 : edit war is is what is going back and
12:53 : forth back and forth
12:55 : um and then yeah it's cool that you can
12:58 : see
12:59 : the you know the authorship of the lines
13:03 : i think let's look at maybe another
13:05 : example
13:09 : um okay so
13:13 : can you tell me what these black bars
13:16 : are
13:16 : here why is it dark
13:20 : for two for two consecutive
13:23 : lines wait does that mean stuff get
13:26 : deleted
13:27 : yeah yeah everything got deleted
13:30 : oh man and on wikipedia that's
13:34 : like a common thing it it's called
13:36 : vandalism
13:37 : like somebody just came in and decided
13:39 : to delete the article
13:41 : wait so is there like people reviewing
13:44 : this kind of stuff
13:45 : or not yeah on wikipedia they actually
13:48 : have a lot of people that are monitoring
13:50 : so yeah so it's okay
13:53 : if somebody just vandalized the site
13:56 : people will notice and then just put it
13:58 : back
13:59 : okay which is what happened
14:04 : yeah so that so you can see this is
14:06 : vandalism
14:07 : and this is i don't know this could be
14:12 : just like a edit like maybe a big
14:15 : section was deleted
14:16 : yeah this is a big section was deleted
14:18 : too like probably from
14:20 : here to here um what i like about this
14:24 : is you can see the movement of the text
14:27 : which which with with the
14:30 : def tool that we use we don't see the
14:33 : movement
14:34 : necessarily oh here like you can see
14:37 : some text is going
14:38 : down and some text is going up here
14:41 : so if if text swapped in position
14:44 : like sometimes you just say hey this
14:48 : function in the bottom i would like to
14:50 : move it up to the top
14:53 : but if you do a diff with git it'll show
14:55 : up as one deletion
14:57 : and one addition to the code right
15:00 : uh-huh
15:00 : so but with this algorithm it actually
15:03 : shows that as a movement
15:05 : well if you changed a lot of stuff in
15:07 : the function
15:08 : like how would that work you not only
15:12 : moved it but you also changed it all
15:15 : right
15:15 : um in
15:18 : a lot of cases this algorithm will also
15:21 : show
15:22 : as a movement
15:25 : this algorithm is interesting in of
15:28 : itself
15:29 : it's called the paul hackeldiff
15:31 : algorithm
15:32 : um there's also the myers-diff algorithm
15:36 : and i just learned about the paul heckle
15:38 : one and
15:40 : there are things about it that i really
15:42 : like um
15:43 : but perhaps we don't have time to go
15:46 : over it right now but one of the things
15:49 : i really like is just this
15:50 : able being able to track movement i
15:53 : think
15:54 : this can also really be useful for
15:57 : for uh reversion controlled projects
16:00 : to to see a visualization like this for
16:04 : a file
16:05 : um yeah you can see where you can see
16:08 : where refactors
16:10 : got done like this is a clearly a
16:13 : refactor
16:14 : over here now this is not for code but
16:18 : for article but the same concept
16:20 : applies like they they uh did a big
16:23 : refactoring
16:24 : over here well actually no they just at
16:27 : the end they just reverted back
16:31 : somebody tried to do a refactor but they
16:34 : abandoned it is what happened
16:40 : yeah yeah that's what happened here
16:44 : uh here
16:48 : there's this temporary edit going on
16:52 : here i don't know like somebody
16:58 : somebody decided to replace some
17:01 : huge three huge sections with something
17:04 : entirely different
17:05 : maybe and then they reverted it back
17:10 : yeah well somebody deleted all the stuff
17:14 : and then replaced and then deleted and
17:16 : then it
17:17 : interesting but it's not a deletion
17:20 : because these movements
17:22 : are going downwards oh that is true
17:25 : well but the blue chunk stopped
17:29 : in the middle uh
17:32 : like on the black uh spot
17:35 : like where there it was black so on the
17:38 : left there was like the blue
17:40 : a big chunk of blue yeah that and then
17:42 : it got removed
17:45 : a while oh yeah this this cyan
17:49 : part yeah it got removed but it kind of
17:52 : came back over here but then you kind of
17:55 : lose the linkage
17:56 : once that happens yeah
18:00 : i think it's pretty interesting to look
18:03 : at
18:06 : yeah and another thing i thought about
18:08 : is like well this is only for one file
18:10 : can we
18:11 : have something like this for the project
18:14 : level
18:14 : can we can you zoom out one level
18:18 : and see a similar visualization
18:21 : um i think you probably can if you take
18:24 : the output of the ls command line
18:28 : and you tell you tell ls to print out
18:32 : the files recursively
18:36 : and then you just dip the output of ls
18:40 : you can probably get a similar view of
18:43 : the whole project so is it at the file
18:47 : level to see like how many files
18:49 : or it are you talking about getting
18:51 : everything
18:52 : in terms of content in a project
18:56 : um i thought about both but but yeah but
19:00 : probably the one that makes more sense
19:01 : is just
19:03 : just have the file listing of the whole
19:05 : project
19:07 : be the content and then dip that
19:10 : at the next versus the next version of
19:13 : all that
19:14 : okay yeah so
19:19 : yeah i i think this could be a very cool
19:22 : tool i i'm kind of motivated to do
19:26 : something about this