Spaceflight Epidemiologist's Grand DAG | Robert Reynolds, PhD S3E01 | CausalBanditsPodcast.com

Join the UBC Causal AI Cluster launch event

In-person & online on October 8, 2026

More on the event here

How do you build a causal model when only a few hundred people in history have ever been your population?

Robert J. Reynolds was brought into NASA to answer questions difficult to settle with a simple experiment: what threatens astronauts health and how to make sure that countering one risk does not increase another one.

To answer these questions, Rob worked with physicians, engineers, physiologists, and statisticians, none of whom speak quite the same language.

What came out of it is one of the most ambitious causal modeling efforts in spaceflight history.

In this episode, we cover:
- How to successfully build a causal model in a multidisciplinary organization
- Why ignorance is important in the scientific process
- What role testable implications play in the development of a causal graph
- What to do when your sample size is tiny and the stakes are high Enjoy the episode!

Recorded on Sep 21, 2026 in New York City, US.

About The Guest

Dr. Robert J. Reynolds is a spaceflight epidemiologist and data scientist working at the intersection of human spaceflight, small-n methodology, and causal inference. At NASA's Johnson Space Center he supported the Human Health and Performance Directorate, applying statistical modeling, systems thinking, machine learning, and network science to decisions about astronaut health and mission readiness. He is also Associate Professor of Aerospace Medicine at the University of Central Florida College of Medicine. He holds a PhD and an MPH in epidemiology and an MS in statistics, and is an Accredited Professional Statistician (PStat) with the American Statistical Association.

Rob delivers the keynote at the launch of UBC's causal AI research cluster, "Graphical Models in AI: Discovery and Application," on October 8, 2026.

Connect with Robert:
- Robert on LinkedIn: linkedin.com/in/RobertReynoldsPhD

About The Host
Aleksander (Alex) Molak is an independent machine learning researcher, educator, entrepreneur and a best-selling author in the area of causality (https://amzn.to/3QhsRz4 ).

Connect with Alex:
- Alex on the Internet: https://bit.ly/aleksander-molak

Links
- Reynolds, R. J. (2026) - "Living DAGs: the future of DAGs in epidemiology " (https://pubmed.ncbi.nlm.nih.gov/41662841/)
- Antonsen, Reynolds, et al. (2024) - "Causal diagramming for assessing human system risk in spaceflight" (https://www.nature.com/articles/s41526-024-00375-7) - Ward, Reynolds, et al. (2024)
- "Levels of evidence for human system risk evaluation" (https://www.nature.com/articles/s41526-024-00372-w)


💠 Alex's Book on Causality
🔷 Consulting and Causal AI Training For Your Team: hello@causalpython.io

YouTube Playlist

Transcript

Copy Transcript Copied

[00:01:47] In medicine
 [00:01:52] nutrition, or in drug safety, detecting patterns is not enough. The next generation of AI looks
 [00:02:00] beyond patterns. It wants to reason, and it wants to reason about cause and effect. At UBC, we study
 [00:02:06] graphical models to answer a fundamental question. What will happen when we intervene?
 [00:02:13] Today, we travel to New York City to talk to Rob Reynolds, a spaceflight epidemiologist who helped
 [00:02:19] NASA-mapped spaceflight risks using causal models and an adjunct associate professor
 [00:02:25] at the University of Central Florida.
 [00:02:28] Ladies and gentlemen, please welcome Dr. Robert Reynolds.
 [00:02:34] Let me pass it to your host, Alex Molak.
 [00:02:37] Welcome to the podcast, Robert.
 [00:02:40] Thanks. Great to be here.
 [00:02:42] Great to have you.
 [00:02:43] I want to start with an idea that I like to think about sometimes.
 [00:02:47] And this is an idea that ignorance has a certain role,
 [00:02:53] plays a certain role in the process of scientific discovery.
 [00:02:56] And I wanted to read a quote to you from Yuval Noah Harari,
 [00:03:00] which goes, it's the following.
 [00:03:02] The greatest scientific discovery was the discovery of ignorance.
 [00:03:06] Once humans realized how little they knew about the world,
 [00:03:11] they suddenly had a very good reason to seek new knowledge,
 [00:03:16] which opened up the scientific road to progress.
 [00:03:20] What do you think about the role of ignorance in the process of scientific discovery?
 [00:03:25] I would largely agree with that quote.
 [00:03:27] I think it's important.
 [00:03:28] For me, it's about saying, what do we think we know already?
 [00:03:31] What are we trying to find out?
 [00:03:33] And then what boundaries does the confluence of those two things create for us in our knowledge?
 [00:03:39] So, yeah, I would largely agree that understanding what we don't know is the driving engine
 [00:03:42] and then trying to develop strategies to work around that is what we're after.
 [00:03:47] You recently published a commentary in the Journal of Epidemiology.
 [00:03:51] American Journal of Epidemiology, yes.
 [00:03:53] Yes. Tell us a little bit more about this commentary and what was the perception in the scientific community of the ideas that you presented there?
 [00:04:03] Well, so the commentary itself was, I thought, fairly simple.
 [00:04:07] was the idea that instead of drawing directed acyclic graphs, using them only to identify
 [00:04:14] causal estimates, and then sort of forgetting that we had them, we should persist those. We
 [00:04:19] should try to share them. We should try to build them into a more systemic explanation of what we
 [00:04:24] think is going on with different research areas. Now, as for the reaction to it, I actually had
 [00:04:29] four reviewers on that simple commentary, which I think is unusual. I don't publish many commentaries,
 [00:04:34] but people tell me that's unusual. So there was a lot of in-depth questions that they had. They
 [00:04:40] wanted to make sure that I wasn't just sort of doing this off the cuff, that I thought about it
 [00:04:44] very carefully. And so after extensive revision of, and then answering those questions, it got
 [00:04:51] published. Now, beyond that, the reaction, I don't know, we'll see. It's gotten a little bit of
 [00:04:56] attention because I've presented some of those ideas at the Society for Epidemiologic Research.
 [00:05:01] people seem interested, but it's a new thing. Most people have just done that. I'm sort of
 [00:05:07] drawing from my one study, and that's all I need this for. I don't think people have tried to think
 [00:05:11] in that broader systemic frame yet, so we'll see how it goes. So if I understand you correctly,
 [00:05:17] the idea that you propose is that we should use the graphical structure not as a static snapshot
 [00:05:24] of a hypothesis that we have, but rather as kind of a database for our structural knowledge about
 [00:05:32] certain processes. Right. It's a visual way to encode all the things we think we know
 [00:05:37] about the entire system of whatever we're interested in. So in my context, it was about
 [00:05:42] human system risk in spaceflight. So we had 29 separate agency-defined risks that people were
 [00:05:49] worried about, things that happen to the human body once you enter the spaceflight environment.
 [00:05:54] And it's all very complex. So trying to understand how those things function just in isolation and
 [00:05:59] they interact with each other was a big challenge. So that sort of naturally extends itself out to
 [00:06:04] other areas. And we've tried this with cerebral palsy, for example, worked well there. Anywhere
 [00:06:09] that you're looking at a complex system then, you can draw one of these pictures and continue to
 [00:06:14] revise it as you learn more. So it's visual bookkeeping on our, at the epistemic state
 [00:06:19] of what we know. What was your process as an epidemiologist? Is that you came, was that you
 [00:06:25] came with your own knowledge and you tried to draw the edges between different variables,
 [00:06:33] so you worked with other people on that. So how did we come up with the DAGs in the first place?
 [00:06:37] Yes, I'm asking about this project that you mentioned, the project with the space agency,
 [00:06:43] the NASA. Yeah, so there the structure was that there was a board that was supposed to
 [00:06:49] gather the evidence about these risks and come up with a risk posture. To do that,
 [00:06:53] there were teams of experts. There was typically a flight surgeon assigned to each team,
 [00:07:00] an epidemiologist, some biostatistical support, and then the subject matter expert scientists,
 [00:07:05] and those were called risk custodial teams. So our approach was go talk to the experts in all
 [00:07:10] these areas, work with them to give them some rules and structure around how to develop their
 [00:07:15] own picture, and then try to combine those later so we would get an entire network out of all the
 [00:07:20] risks. How is the process? Can you tell us a little bit more about the process? How you worked with
 [00:07:25] these people? Was that something like you just go and run an interview with someone and then just
 [00:07:34] write things down and then you go to another person or was it something that was a little bit
 [00:07:39] more interactive? A little more interactive and more extensive. We took essentially two years to
 [00:07:45] work our way through all of the teams. And it was us sitting down at the leadership team from the
 [00:07:51] board and working with the experts and saying, all right, let's start with the hazards on one
 [00:07:56] side of the canvas, the mission level outcomes, which are agency defined on the other. And let's
 [00:08:02] start thinking about how the hazards create the risks. And so we stepped them through the process
 [00:08:06] of saying, all right, you've written this paper over here and you said these things are important,
 [00:08:10] let's get them on the canvas. Now, how do they wire together with the rest of it? And so it was
 [00:08:14] this interactive back and forth of helping them find the right level of detail, understanding
 [00:08:18] how to represent certain things that were maybe not straightforward, until we got to
 [00:08:23] 29 of those pictures.
 [00:08:24] And what was next?
 [00:08:25] Well, then we said, all right, if we're going to put them together, we need to make
 [00:08:30] sure that when they're talking about the same factors, that they're using the same terminology.
 [00:08:35] Because if you've ever worked with DAGs, which I think you have, you have an edge list in
 [00:08:39] some form, and in order to make these networks combine or gel together, it's
 [00:08:44] going to be the nodes that do it. Now they may share edges, but that's still a
 [00:08:48] function of the nodes. If they don't have the same nodes called the same things,
 [00:08:51] they won't mesh together correctly. So terminology harmonization was the next
 [00:08:56] big step. For example, there was 11 different ways that people were talking
 [00:08:59] about a change in gravitational force. So some were calling it altered gravity,
 [00:09:04] some were saying microgravity, zero gravity, accelerated gravity. So we had
 [00:09:08] bring all those under one umbrella where we just said all right we're going to call these things
 [00:09:11] altered gravity and that's going to cover any situation where you're not in earth normal
 [00:09:16] gravity that's what allowed all those 29 networks to gel together into one big one
 [00:09:21] and that had 400 plus nodes and 1500 plus edges so it's a picture that no one could have drawn on
 [00:09:28] their own but once we get people to draw the little bits they understand it becomes a coherent
 [00:09:32] explanation for the entire thing two years the entire process that was just to get the initial
 [00:09:37] pictures, yeah. And then the harmonization added a couple more months, and finally we were able to
 [00:09:42] put something out in year three. What was that? That was the big network, yeah, and the formal
 [00:09:47] pictures for the smaller networks. Those would come to the board, so that's another important
 [00:09:53] part of this expert elicitation process that we underwent. Wasn't just the small teams, but also
 [00:09:59] the broader community at NASA that was human health and performance. So they would bring their picture
 [00:10:05] the board, say, here's our explanation. And there would be a review period in which anybody in the
 [00:10:10] HHPC, the Human Health and Performance or HHPD directorate, would be allowed to make comments
 [00:10:17] and suggestions. And so there was some, in that way, crowdsourced corrections, if you will, to each
 [00:10:23] of the diagrams. People from other areas saying, well, we think about that too. And we think it
 [00:10:28] wires like this. And so there were some times when we brought more than one group to the table and
 [00:10:32] them hash it out and say, how do we think this works? If you guys are saying one thing and you're
 [00:10:37] saying the other, they can't both be true. So how do we reconcile that? So there was a lot of round
 [00:10:42] robin in that sense of community participation and group to group talking and really just making
 [00:10:48] sure all our explanations were coherent within and between each other. So it sounds like you
 [00:10:53] like to build like a very open space where people were able to look at the results obtained by
 [00:11:02] others and maybe criticize or assess them and then get into some interactions and so far.
 [00:11:11] Yeah, that was the design of it. We wanted everybody to be included so that no one would
 [00:11:15] say, well, I'm not going to listen to them because I didn't get to put my input into their diagram
 [00:11:20] and I don't think they did it right. Part of our goal was communication. We were trying to have
 [00:11:25] engineers speak to program managers, speak to life scientists and epidemiologists and
 [00:11:29] biostatisticians and flight surgeons. Sometimes when they were talking about the same things,
 [00:11:33] they would talk about it in different ways because it was a disciplinary perspective.
 [00:11:38] So that data harmonization was more than just the mechanics of the graph. It was also getting
 [00:11:42] everybody on the same page about what do we really mean when we say altered gravity, for example.
 [00:11:48] What do you think were two or three most important aspects of this process that made it successful in the end?
 [00:11:57] So I think that that collaborative approach, the crowdsourcing was really key, and then getting them to agree to that terminology.
 [00:12:05] So just because you identify things you think are the same across those groups, you have to bring them back to the table and say, are you okay with this term?
 [00:12:12] Does this capture the essence of what you're thinking about?
 [00:12:15] And then make sure you write a definition that clearly states those things, too, so that later, when the people who wrote those nodes and wrote those terms aren't at the table, you can refer back to that definition and still be on the right track as to what it's supposed to mean.
 [00:12:29] And then some of the subsequent work, we're seeing that I think that paid off, because people who've never worked those networks before are able to see those definitions of the nodes and understand what the intent is there.
 [00:12:39] It sounds like you devoted a lot of effort. You put a lot of work into building this
 [00:12:45] huge structure. You said 450 nodes?
 [00:12:48] Something like that. Yeah. I forget the exact count, maybe 470 something.
 [00:12:53] What was the goal of building this structure?
 [00:12:57] Well, again, when I was brought on to NASA, I came in as a visiting data scientist. So I had
 [00:13:03] a two-year appointment and there wasn't a list of deliverables for the position. It was more about
 [00:13:08] be a capability that can help the agency do things better. So by first week there, people came and
 [00:13:14] they put me in a room and people came and said, here are the problems I have. Maybe you can help
 [00:13:18] me. This was one of them. The chair of the board at the time, Erik Antonsen, said, I have this
 [00:13:23] portfolio of risks here. We know that they interact. And we know that if we try to develop
 [00:13:28] countermeasures for one, it could change something in the other. But we don't really have a way to
 [00:13:33] see that. So he said, anything you could think of that might be helpful there would greatly
 [00:13:37] appreciated. So I started ruminating on the problem. While a network map seemed like the
 [00:13:42] obvious solution, the basis for drawing it was the challenge. And so it turned out that causation
 [00:13:48] made a great anchor for the whole thing, because then we had a flow, again, from the spaceflight
 [00:13:53] hazards out to the predefined outcomes that the agency cared about. And we could start thinking
 [00:13:58] about how do we disrupt that flow, which is really what we want when we talk about countermeasure
 [00:14:02] development. We want to do something exogenous to the network or change something that's already
 [00:14:07] in the network that will prevent an astronaut from having something debilitating physical or
 [00:14:12] cognitively and will not disrupt the missions, therefore, essentially, or lead to bad long-term
 [00:14:21] health consequences because there are some outcomes that they track there as well.
 [00:14:24] Many people working with DAGs or people who learn about DAGs for the first time in their
 [00:14:31] career, think about the idea of nodes that we are not able to observe. Maybe they are hidden
 [00:14:38] confounders, or maybe they play other roles in the map, maybe unknown roles, right, from the
 [00:14:45] perspective of someone developing a DAG. They don't understand even what role some kind of an
 [00:14:50] unobserved node could play. How did you approach this challenge in your work? Well, I think there
 [00:14:57] was, first of all, the explicit recognition that the DAGs we were drawing were not the right
 [00:15:01] answer. They were a starting place. And part of the goal was to get all of those assumptions onto
 [00:15:07] the canvas so we can start testing them and say, where does this DAG seem to represent reality well?
 [00:15:13] Where does it seem to be falling short? Where it starts falling short is the locations where
 [00:15:18] perhaps there are things that are not on the canvas that ought to be. So that gives you a
 [00:15:22] clue that something's not quite right over in this corner. Maybe we're missing something. It could be
 [00:15:27] missing connections on the things you already see, or it could be those sort of latent variables
 [00:15:32] that either you didn't think to include or you didn't think were important enough to include.
 [00:15:36] So the point was always revision and testing and moving towards a DAG that could be serviceable
 [00:15:43] towards the risk reduction objectives of the agency. Now, if you move that out to a more
 [00:15:47] applied science context anywhere else, you may have slightly different goals, but I think it
 [00:15:52] could work the same way. If you're looking at disease prevention, you might be saying,
 [00:15:56] what is the sort of precursor to the disease that we need to address in primary prevention,
 [00:16:01] for example, lower the incidence of this condition.
 [00:16:04] Or if you're looking at secondary prevention, it's much the same.
 [00:16:08] You're saying, where in the network could we intervene that'll keep this from getting worse
 [00:16:11] once somebody's diagnosed with it?
 [00:16:13] So I think the principles carry over.
 [00:16:15] It's just a matter of what is the threshold you're shedding and what's your overall strategy.
 [00:16:19] Other latent variables you may know about, you just can't measure them.
 [00:16:24] So, there are techniques, of course, that can be borrowed from psychology and economics
 [00:16:29] where you can deal with that.
 [00:16:31] That's more of a mathematical approach, I think, as far as, you know, is it important
 [00:16:35] enough to be part of the structure?
 [00:16:37] Sometimes yes, sometimes no.
 [00:16:39] You just got to get in there and start working with it and bring data against it and find
 [00:16:43] out.
 [00:16:44] What was your approach when it comes to filling in the structure with the data, with
 [00:16:50] the information from, you know, actual observations?
 [00:16:53] Well, when you start drawing expert elicited DAGs, I think the first criticism you'll run
 [00:16:58] up against is how do you know it's right?
 [00:17:00] And so for me, there's a couple of ways to know that it's right enough for now to get
 [00:17:04] us started.
 [00:17:05] Probably three at least.
 [00:17:06] One is that socialization with other experts.
 [00:17:09] Let the people who have rocks to throw, throw them, see what changes that implies about
 [00:17:13] the DAG.
 [00:17:14] Get through that process, you start bringing data.
 [00:17:18] First thing to notice is that every arrow on an expert elicited graph is really a hypothesis.
 [00:17:23] making an assertion that A causes B. In some cases, that'll be, you know, painfully obvious
 [00:17:29] and probably doesn't need a whole lot of new evidence for it. Other cases might be somewhere
 [00:17:34] in the middle. For the spaceflight context, we might notice something that's normal Earth
 [00:17:38] physiology, but then we wonder, will it happen in space too, or will it happen the same way?
 [00:17:44] So there, you might hit the literature and start saying, here's the Earth-based stuff,
 [00:17:48] here's what we've seen in analogs, here's what we've seen in space, and try to build a level
 [00:17:52] of evidence around the arrow.
 [00:17:55] Now the third and final, or maybe not even final, but at least the only one I know of
 [00:17:59] for now, a way to do it is to bring information about multiple nodes at once, data sets from
 [00:18:05] studies, and test the structure of the DAG.
 [00:18:09] So you can look at the conditional independencies and test those all together.
 [00:18:15] And that's where if this DAG is correct, then A should be independent of B when we control
 [00:18:19] for C, that sort of thing.
 [00:18:21] You can test marginal dependencies, marginal independencies.
 [00:18:24] Those provide you with less information, less informative.
 [00:18:27] But overall, there are predictions that the DAG makes that you can test.
 [00:18:31] And so that's, at least at this point, the strategy I see.
 [00:18:36] Build up expert-elicited agreement about the structure.
 [00:18:40] Bring the literature where you need to to show that the errors are true, at least in
 [00:18:43] isolation, and then try to bring data to show that the structure seems to hold.
 [00:18:48] you've exhausted existing data, then it can start to suggest the studies you might want to do
 [00:18:52] to fill in those parts of the graph about which you are unsure.
 [00:18:55] When you talk about things like space light epidemiology, my thinking immediately goes in
 [00:19:02] the direction of looking at different levels of system description, right? So we can talk about
 [00:19:09] maybe molecular level or physiological level. Then we can talk about the level of symptoms,
 [00:19:16] like medical symptoms. Then we can talk about physics. We can talk about time-dependent change
 [00:19:25] and so on and so on. What is your thinking here and what was your approach to making sure that
 [00:19:33] these different levels, assuming that you used descriptions at different levels,
 [00:19:39] were congruent with each other? So I think that DAGs are great because you can represent all of
 [00:19:44] those things in the same picture. You don't have to have a separate physics
 [00:19:48] tag from a separate cellular process tag. You can map the effect of gravity on
 [00:19:53] cells and then ride that all the way up to the organism level or the noticeable
 [00:19:57] symptom level if you want to. I think the key is just to be very explicit about
 [00:20:00] what it is that the node represents at any point in your graph. You want to
 [00:20:06] make sure that you represent it in a way that can account for all the
 [00:20:09] possibilities. So one of the things we had our teams take note of is don't say
 [00:20:14] an increase in something like urinary calcium. Just say urinary calcium level, because it could
 [00:20:20] be high, it could be low, it could go up, it could go down. If you frame those variables neutral,
 [00:20:25] then you can account for that in the actual parameterization of that variable rather than
 [00:20:30] in its name. So I think that's part of it. Understanding which parts are the structure
 [00:20:34] and which parts are the math is important too. That helps it keep clear in your head how to draw
 [00:20:40] things. Some of the relationships people get hung up on are really about the joint probability
 [00:20:45] distribution in the node rather than the structure itself. So if you can keep your nodes and your
 [00:20:50] arrows separate from your math, at least initially, I think you'll draw a clearer picture and
 [00:20:56] disambiguate your own thinking. In the process, have you ever experienced something like
 [00:21:01] getting feedback from another person saying, oh, we already researched that and your representation,
 [00:21:07] where something might be wrong there because it's incongruent with, you know, what we see in our
 [00:21:14] theory or in our research. We had a little bit of that in our NASA teams.
 [00:21:22] There was an example where one team thought you couldn't measure motivation,
 [00:21:25] and the psychological team said, yes, you can, and we'll show you how. And so that opened a
 [00:21:31] whole new realm of possibilities to where, okay, motivation can now be mapped into exercise
 [00:21:36] physiology concerns. Because yes, you can actually measure that if you want to and understand how it
 [00:21:41] plays on someone's desire to do exercise, for example. That's very interesting. Yeah. So I guess
 [00:21:48] the summary of that would be that sometimes those aren't the conflicts they appear to be.
 [00:21:52] They're perhaps just disciplines not understanding what the other one does. So that's why you bring
 [00:21:57] them to the table and try to hash it out and say, you know, why did you put that on there? What does
 [00:22:02] mean to you? And you might discover that there is something like motivation that can be measured that
 [00:22:08] you didn't think could be. You talked a little bit about this idea of testable implications.
 [00:22:13] There might be certain implications when we look at the graph structure, for instance, right? You
 [00:22:19] told us about conditional independencies. That's one of testable implications. Another one could be
 [00:22:25] that if we have an edge between two nodes and maybe there is some external research about this
 [00:22:32] particular connection, we are expecting to see some effect between those two nodes, right? If
 [00:22:38] there is no effect in external research and we believe that this research is valid in terms of
 [00:22:45] the design and analysis and so on and so on, then that will be probably at least a moderate piece
 [00:22:51] of evidence against the hypothesis that there should be an edge in our graph representing
 [00:22:57] a relationship between these two given nodes.
 [00:22:59] Are there any other methodologies that you used to either gather evidence for or against
 [00:23:10] the hypothesis or the edges in your graph?
 [00:23:13] Well, so what you're suggesting is that if we're unsure about edges, we could retreat
 [00:23:17] to the literature and say, does the weight of evidence at the time suggest that we ought
 [00:23:21] to draw that arrow or not. I think that's a valid strategy. But when you're talking about expert
 [00:23:26] elicited DAGs, people aren't going in that blind usually. And there could be some details about
 [00:23:31] which they're unsure, but that's why you build a multidisciplinary team. Because if the exercise
 [00:23:36] physiologist doesn't quite know the answer, perhaps the physician does. At least enough to
 [00:23:40] confidently or not put in an arrow or omit it. But in general, you should go back to the literature
 [00:23:46] anyways. We're running a program right now, or a project rather, through what's called an analysis
 [00:23:52] working group. These are sponsored by NASA Ames, and they're citizen science groups. So we're
 [00:23:58] taking one of NASA's DAGs, the renal stone formation DAG, and we're doing exactly that.
 [00:24:04] We're looking at all of the edges in that graph and saying which one of these are
 [00:24:08] canonical physiology, things that nobody would ever argue with and are super well established
 [00:24:12] for a century or more. We'll set those aside because those you can go find the
 [00:24:17] textbook references and that's probably good enough. Another set that says these
 [00:24:22] are by definition or by design. So where NASA mapped in things like vehicle design
 [00:24:27] causes the medical capability, that's just a fact. If you don't bring it with
 [00:24:32] you, you don't have it, you can't use it. So that's a by definition or a by design
 [00:24:36] kind of relationship. Then there's the other ones which is those relationships
 [00:24:40] are perhaps not canonical. They're not by design. So we need to bring some evidence to say,
 [00:24:45] here's why we believe in these arrows. So we're doing that. And what you get out of that process
 [00:24:51] is what we call a level of evidence assessment. And so we developed a system to go with the NASA
 [00:24:56] DAGs where it goes from speculative, weak, moderate, strong. And this is just through
 [00:25:02] progressive satisfaction of the Hill guidelines. So the speculative node or speculative edge
 [00:25:09] would have simply analogy and temporality we know that this causal relationship we think the cause
 [00:25:15] comes first and the effect comes later and we can point to something else and say we think it looks
 [00:25:20] like that we think it works the same way as that thing over there so you're not always in the
 [00:25:25] situation where it's like we're totally sure about this edge or we're totally unsure there's room
 [00:25:30] there to be tentative about it and build towards certainty or reverse course later and say you know
 [00:25:35] what? We looked at the evidence. Doesn't look like this is true. We're going to take it off."
 [00:25:40] You brought us back to the beginning now, in a sense. The topic of ignorance, right?
 [00:25:46] The role of ignorance in the scientific process. My understanding of what you're saying here is
 [00:25:50] that it's important to distinguish between situations where we are relatively sure that
 [00:26:00] something exists or there's a relationship between two different variables versus a situation, for
 [00:26:06] instance, where we are rather unsure about that, right? Because this leads us to different decisions
 [00:26:13] and different actions down the road. Right. And so in the context of risk management, that's
 [00:26:18] always top of mind because what you're doing is saying, I have to avoid bad things happening
 [00:26:24] and I can't wait around to be 100% sure. So what do I do today that works off the knowledge I have
 [00:26:31] that can actually accomplish the goal? So we're always going to be in a state of flux.
 [00:26:36] I can't imagine anything other than the most trivial graph ever being one where somebody says
 [00:26:41] I'm totally sure about every arrow and every note on this picture. Most of the time we're going to
 [00:26:45] be operating in an environment of some level of uncertainty. So the key is to be very upfront
 [00:26:50] about that and get that out in the open so we understand what we don't understand.
 [00:26:54] And that's where we can focus our efforts and continue to refine the explanation.
 [00:27:00] So to me, that's what living DAG concept is all about, which was the subject of the commentary
 [00:27:04] in AJAE.
 [00:27:05] It sounds to me that if we have a reliable assessment of our own ignorance, we can also
 [00:27:16] then weight our decisions by this ignorance.
 [00:27:19] So for instance, something that is, we are pretty certain about something and maybe this
 [00:27:24] is related to some very high risk, could be prioritized versus something that we are very
 [00:27:31] uncertain of and maybe there is moderate level of risk.
 [00:27:35] I think this has a common sense analogy.
 [00:27:39] If you've ever made a mistake, and I can't imagine you've done that, but I've done it
 [00:27:43] once or twice, you think back and you say, oh, that was so stupid, why did I do that?
 [00:27:48] But if you're kinder to yourself, you think back to what was my state of knowledge at that moment.
 [00:27:53] And you typically make the best decision you can, given what you know in the moment.
 [00:27:58] So waiting on something like our uncertainty is a formalization of that, in a way.
 [00:28:03] Because if we say, I don't really understand what's going on over here, but I do understand this part really well.
 [00:28:08] Let's focus on the things we do know well and see if we can affect the change we're looking for.
 [00:28:13] Then you're doing exactly that thing.
 [00:28:15] Sometimes it'll work out, sometimes it won't.
 [00:28:17] That's just the way it goes, I suppose.
 [00:28:20] But there's also a flip side to that, where if you're saying, I'm going to be very intentional
 [00:28:25] about gathering new evidence, then you could almost inverse weight it so that all of your
 [00:28:30] inquiry heads to the part of the network you don't understand well.
 [00:28:33] And you would say, all right, if this is true over here, that node would be really consequential,
 [00:28:39] so I want to be sure that that relationship is true.
 [00:28:42] Is that something you used in your work with NASA as well?
 [00:28:47] something we've proposed. Once we got these DAGs put in, it seemed like there was a million
 [00:28:52] directions we could go. Unfortunately, I don't have the time to do everything, but we're trying
 [00:28:58] to do the things that we think are the most consequential. One of those would be to figure
 [00:29:02] out, yeah, how do we use this structure, even in absence of detailed data, to try to understand
 [00:29:08] the ways to lower our risk. If you wanted to become a program manager and use that other part,
 [00:29:15] that would be the opportunity. Use structural metrics, weight by uncertainty, and that would
 [00:29:21] lead you towards the things you ought to be investing time, money, and other resources into.
 [00:29:26] When I'm thinking about your time at NASA, and in particular, filling in the structure with
 [00:29:33] data, with actual observations or results of experiments, one thing that comes back to me is
 [00:29:40] the sample size that is available for you? Because I'm not an expert in that, so I don't know
 [00:29:47] really how many astronauts we had and what level of, you know, how many variables and with what
 [00:29:55] kind of precision we were able to record the responses to the circumstances with. But it
 [00:30:02] seems to me that we don't have that many people that were part of a space mission.
 [00:30:07] Yeah. In epidemiological terms, we have virtually nothing. If you look at large studies of chronic disease or whether or not cell phones cause brain cancer, you can get large European cohorts that have millions of people followed for a decade or more. That'll generate 10, 20 million person years worth of observation.
 [00:30:27] With spaceflight, we've had around 700 people that pass the Karman line and qualify as having gone to space.
 [00:30:34] With the U.S. Astronaut Corps through the end of 2022, I have about 12,000 usable person years.
 [00:30:40] And massive selection bias because NASA selects for really healthy people.
 [00:30:46] And so we have low numbers of actual outcomes that you might care about, chronic disease, for example.
 [00:30:51] So it's just really a difficult inference environment from a traditional epidemiological perspective.
 [00:30:56] So that was part of this, was can the structure help us out with the fact that we don't have a lot of data to use?
 [00:31:02] Now, one of the goals was always to turn our DAGs into a Bayesian network,
 [00:31:06] but that, of course, requires a decent amount of data.
 [00:31:09] So we're not going to get that from spaceflight alone, at least not anytime soon.
 [00:31:15] As we see the rise of commercial spaceflight, we'll get more, but it still won't be enough.
 [00:31:20] So spaceflight has always relied heavily on analogs and terrestrial similarities, occupational
 [00:31:27] cohorts and submarine crews and Arctic winterovers, Antarctic winterovers rather.
 [00:31:33] And part of that then becomes a translational challenge because now we're looking for specificity.
 [00:31:39] We're looking for seeing the actual effects in people in context, but we don't have the
 [00:31:44] people in context to do that with.
 [00:31:47] When I think about the sample size, estimation is one challenge that you mentioned.
 [00:31:53] Another one that comes to my mind is related to positivity.
 [00:31:55] So the idea that for valid causal inference, we need to have information about both potential
 [00:32:07] outcomes.
 [00:32:08] Right?
 [00:32:09] People, this type of person with treatment and without, and another type of person also
 [00:32:15] with treatment and without.
 [00:32:17] What is your thinking about positivity in this context?
 [00:32:20] I've tried to do some thinking about this.
 [00:32:23] There's situations where you can start to set up a traditional epidemiological look
 [00:32:28] at things.
 [00:32:30] You might say, what's the effect of being a Russian and Soviet cosmonaut versus being
 [00:32:34] a NASA astronaut?
 [00:32:36] Does that lead to chronic disease differences?
 [00:32:39] And you could whiteboard out some reasons why it might.
 [00:32:42] decisions in the design of spacecraft, different mission profiles, different general population
 [00:32:49] background, incidence rates, and mortality rates from the different nations. But then you go to
 [00:32:54] try to run it in a traditional sense, like maybe you want to do a propensity model, and you run
 [00:32:59] right up against this idea that positivity is absolutely violated, and it always will be,
 [00:33:04] because the chance of a Soviet cosmonaut being assigned to the NASA astronaut corps instead
 [00:33:11] is truly and really zero right in the 60s at the height of the cold war that was not a possibility
 [00:33:18] so even though you could try to do a propensity analysis and build a group where they're the same
 [00:33:24] age and they're the same height and weight and blood pressure and all this and they look very
 [00:33:28] similar medically and demographically there still is no true positivity there so for me that's
 [00:33:35] something i'm still thinking about is is there such a thing as ignorable positivity can we sort
 [00:33:40] brush that aside just knowing that it's not real and what would be the implication of it for the
 [00:33:45] analyses that we run. I don't know the answer to that yet. If anybody else does, find me.
 [00:33:51] That's interesting. Now, because when you think about it,
 [00:33:55] what does it mean being American or being Russian, really?
 [00:33:58] Well, there's differences besides just language, obviously. Place, of course,
 [00:34:05] has been a popular thing for epidemiology to think about since the beginning. And that can
 [00:34:10] be inclusive of climate and other environmental factors. There's diet. There's philosophy about
 [00:34:18] how to live your life, if you will. And then there's all those other factors that I mentioned
 [00:34:23] before, different design in spacecraft, different mission profiles. So an example, this is anecdotal,
 [00:34:29] but people tell me that the ISS is in the orbit it's in because the partnership with Russia means
 [00:34:35] that they wanted it so that it would pass over Russia and they could do their mission control
 [00:34:39] easier, but that makes them fly through the South Atlantic anomaly. So you get more radiation
 [00:34:43] in that orbit than you would if you intentionally avoided it, but avoiding it wouldn't bring it over
 [00:34:49] Russia. So there's trade-offs and the Russians would have made those trade-offs unilaterally
 [00:34:55] for their program. They would have had their spacecraft fly over their mission control center.
 [00:35:00] So that means they probably got more radiation, for example. So these things have consequences,
 [00:35:06] but it's still one of those questions that even if you don't really care about what it means to
 [00:35:11] be Russian or if it creates differences with American astronauts, it's worth thinking about
 [00:35:16] positivity in that frame. It gives us something to latch on to because there may be a situation
 [00:35:22] where we think about spaceflight exposure rather than nationality and positivity will be equally
 [00:35:27] violated. And what is that going to mean? I don't know yet. It seems to me that the work you've done,
 [00:35:34] The entire framework that you created in the process of building this DAG could be also applied elsewhere.
 [00:35:43] In the beginning, you mentioned your new research focused on cerebral palsy.
 [00:35:48] Yes.
 [00:35:49] Do you see a translational potential, first, for the framework that you created during your work at NASA?
 [00:36:00] and second, some particular aspects of maybe of the DAC, of the structure, translating them or
 [00:36:08] applying them elsewhere. Yeah, I think that was part of the reason for doing it. I had two thoughts
 [00:36:15] in mind. The first was cerebral palsy is really complex and I don't think I understand it all that
 [00:36:19] well and it would sure be great to have a picture. But simultaneously it becomes, is this framework
 [00:36:25] portable? If it worked for spaceflight, would it work in developmental disabilities? So I reached
 [00:36:30] out to some colleagues that are involved in the epidemiology and treatment of cerebral palsy.
 [00:36:35] And we took two years to go through methodically in very similar fashion, build out a etiology DAG
 [00:36:41] for CP. So we got clinicians in Europe, Australia, Asia, and some folks in the United States here
 [00:36:49] and took two years to just work through it. We met monthly. We said, let's start with what we
 [00:36:56] think are the exogenous inputs on one end and we'll put CP on the other and in the same way
 [00:37:01] we'll wire it together. How does one thing lead to the next? They were very meticulous, that group,
 [00:37:06] especially our Australian friends, in not only writing a DAGtionary and defining the terms really
 [00:37:11] well but then bringing the literature that they already knew about that would support those nodes
 [00:37:16] on the canvas. So right out the gate we had good literature support on that DAG and we're just
 [00:37:22] getting to the point now where we can start to look at the structure of it, pick the tires
 [00:37:26] a bit, understand it as a network object, and see where it takes us from there.
 [00:37:30] I remember you told me that in your work you were not only looking at the graph from
 [00:37:36] the causal perspective, but also through the lens of network science.
 [00:37:41] Right.
 [00:37:42] I'm a bit of a generalist.
 [00:37:43] So I started out in psychology as an undergrad and was fascinated with the idea of structural
 [00:37:48] equation models, path analysis, and this measuring of latent variables, something we touched
 [00:37:53] on a little earlier.
 [00:37:54] So when I saw a DAG for the first time in epidemiology, I immediately said, hey, that
 [00:37:58] looks a lot like the SEM path diagrams, with some important differences, of course.
 [00:38:03] But fundamentally, they were trying to tell us the same thing, the relationship between
 [00:38:06] variables, what causes what.
 [00:38:09] So when I look at DAGs, I think, yes, we can get causal estimates, we can look at confounding,
 [00:38:14] we can design our analysis.
 [00:38:16] But also, that could be the basis of a path model or a structural equation model.
 [00:38:20] And that could be the basis of a Bayesian network.
 [00:38:23] So that same structure, to me, is very portable into different contexts because it's doing the same work.
 [00:38:28] It's providing the context, the structure, the causal explanation upon which the math then gets built.
 [00:38:35] So it's a portable scaffold, if you will.
 [00:38:38] What was the biggest challenge for you in this entire program?
 [00:38:43] I think trying to keep my eye on the prize throughout the process.
 [00:38:46] It was, you know, when you set out to do these things and it takes two or three years to get to an end product, you really got to keep in mind, what am I trying to get to in the end?
 [00:38:58] And so going through all the rounds of revision, meeting with different groups, getting them to draw pictures, harmonizing terms, and then finally putting it together.
 [00:39:07] It's like, ah, that's what we were trying to get to.
 [00:39:10] So just having faith that you can get there when you've never done it before, I think was the toughest part initially.
 [00:39:15] What was the most valuable, in your assessment, outcome of the program so far?
 [00:39:23] Well, for NASA, it was that communication piece.
 [00:39:26] The moment when one group looks at another's DAG and you can sort of see the light bulb
 [00:39:30] go on and say, that's what they meant by that, now I get it.
 [00:39:35] And then they start to see how their risk folds into that, and they start in real time
 [00:39:39] sort of understanding the interchange of all the 29 individual isolated risks into the whole system.
 [00:39:46] So that was very satisfying, actually.
 [00:39:49] That's great to hear.
 [00:39:50] When we think, we discussed briefly this idea of testable implications, right?
 [00:39:55] That we can...
 [00:39:56] Structure implies math, yes.
 [00:39:58] Yes, and we can stress test certain hypotheses using these implications.
 [00:40:02] In the machine learning community, people often think about predictive power as an indicator of model quality.
 [00:40:13] Did you have an opportunity to use the structure and the, let's say, evasion network or maybe another model built on top of the structure to generate some predictions and validate them against reality?
 [00:40:29] Not so far. We've been more down in the weeds with following the path of trying to say that
 [00:40:36] the DAG is good before we start using it. So we did a small demonstration project where we used
 [00:40:41] data from rats that had either been tail suspended or sent to space. And then the researchers were
 [00:40:49] looking at their bone structure to look for changes in mass and architecture. So that provided
 [00:40:53] us several measurements that we isolated out basically the changes in bone from the greater
 [00:40:59] bone fracture risk. It's a very select set of pathways where you're looking at resorption
 [00:41:04] and formation, and that provided us enough to test that structure. So that was just us sort of
 [00:41:11] figuring out how to do it. Then more formally with NASA, we did a project over the last couple of
 [00:41:16] years where, again, we were using animal analog data and looking at HFBP or human factors and
 [00:41:23] behavioral performance outcomes. So this is changes in cognition and potentially affect
 [00:41:29] based on, again, being exposed to spaceflight or tail suspension or something that somehow
 [00:41:36] mimics some of those exposures, radiation in a lot of the cases. The idea being, does the rat
 [00:41:42] shows something that changes in its behavior that's indicative of changes in its cognition.
 [00:41:48] So again, just testing that structure. The second time through, we came up with a much more formal
 [00:41:53] process, and we're going to be documenting that and putting it out. Everything from how do you
 [00:41:58] pick your variables, how do you match them onto the DAG, when to meta-analyze or use them straight
 [00:42:03] up. There's a whole lot of decisions that go into that. So we're still in that, you know,
 [00:42:07] demonstrating that the DAG represents reality well sort of phase.
 [00:42:12] But we would absolutely love to get into trying to pour some real data into the DAGs
 [00:42:17] and then see how well they predict things.
 [00:42:20] I keep my fingers crossed.
 [00:42:22] Yeah, and I'm not the only one out here. We can all do it.
 [00:42:25] Find a DAG you love and try to do it yourself.
 [00:42:27] What would be your advice for someone who would like to build a similar quality DAG for their use case,
 [00:42:36] whether it's in research or business?
 [00:42:39] I think the key is getting more people involved.
 [00:42:42] Don't try to do it in isolation.
 [00:42:44] If you feel like you know a lot about something,
 [00:42:46] you could draw an initial version,
 [00:42:48] but then you need to start roping colleagues in and say,
 [00:42:50] where did I get this wrong?
 [00:42:51] What am I missing?
 [00:42:52] What have I forgotten?
 [00:42:54] And through that, you're going to start to get a much richer DAG
 [00:42:58] because people are thinking about things you're not thinking about
 [00:43:01] and you're thinking about things they're not thinking about.
 [00:43:03] So bringing those thoughts together leads to a fuller network that captures more of the explanation.
 [00:43:10] And that doesn't just mean in your institution.
 [00:43:12] Reach out to people you know internationally. Talk to people at conferences.
 [00:43:15] Say, hey, we're all working on this from different angles.
 [00:43:19] Have we thought about putting together a unified picture that can allow us all to work better together?
 [00:43:23] And that's what I'd love to see happen is that around societies and conferences,
 [00:43:28] people who do the same thing come together and draw a unified picture like we did with CP.
 [00:43:33] that's now being used by all the researchers in our little consortium and hopefully people
 [00:43:39] outside of that to start saying, we don't know enough about this part of it. Let's start
 [00:43:44] doing experiments or gathering registry data or whatever is appropriate to learn more about it.
 [00:43:50] That's really beautiful. I think we often, or at least people working, you know, maybe on the
 [00:43:56] method side and focusing on, let's say, identification results and so on, we might
 [00:44:00] sometimes forget about the value of the communication aspect of a device like this, of a DAG, right?
 [00:44:08] That it can really help people, synchronize people that work in different places on maybe
 [00:44:14] slightly different aspects of the same problem.
 [00:44:17] Right.
 [00:44:18] I would love to elevate DAGs from just something we take for granted, where we're going to
 [00:44:23] do some sophisticated computer science action on it.
 [00:44:26] So we just say, let's assume this is a good DAG, and now it's all about the math.
 [00:44:30] to bringing it back to a more anchored reality based thing where we're saying this is what we
 [00:44:35] think is a valid explanation. This is what's really happening. Now let's do our math and see
 [00:44:40] what we can learn from it. Or in epidemiology, let's not just map our data set. Let's not just
 [00:44:46] pull out an estimate and figure out what's confounding here, but let's persist this thing.
 [00:44:50] Let's pass it to the next group. Let's build it together. And that way it book keeps all of our
 [00:44:55] knowledge collectively. Your commentary paper that we started with, the future of DAGs in
 [00:45:01] epidemiology? Living DAGs, the future of DAGs in epidemiology, yes. What's the future? Well,
 [00:45:08] my hope is that the future is exactly what the commentary talks about. We stop viewing DAGs as
 [00:45:14] just some kind of analytical tool that we use in the beginning and then throw them away, forget
 [00:45:19] that we had them, and we start persisting them, sharing them, and working on them collaboratively.
 [00:45:26] Tenet has that well-known review where he looked at the use of DAGs in epidemiology, and I can't
 [00:45:31] quote the numbers for you, but there was a sizable chunk that didn't even put their DAGs in the paper.
 [00:45:36] They just mentioned that they had one and then failed to produce it. Another set that used it
 [00:45:41] for estimation and then never used it again. I think that's probably most of them. So for me,
 [00:45:46] an ideal world would be one where we start thinking not just about what's the DAG for my
 [00:45:51] study, but how does this system work in general? And then once you have that, you can extract out
 [00:45:57] of it the more specific DAG for any study, really. And you can enhance it by adding design variables
 [00:46:03] that matter in the timings and all the things that allow you to get to a causal estimate if you need
 [00:46:08] it. In the ideal world, if people read your paper, what would be the outcome you'd hope for? I suppose
 [00:46:16] just that realization that whatever problem you're working on other people
 [00:46:20] are probably working on it too and that there's likely assumptions you're holding
 [00:46:25] that they're not holding and vice versa and that if you would start working
 [00:46:29] with those people if we all started collaborating we could get all of those
 [00:46:33] assumptions onto the table and hopefully start doing better science because of it
 [00:46:37] we can either reconcile them and understand that we weren't actually in
 [00:46:42] contradiction in the first place or start doing the research necessary to
 [00:46:46] figure out which of the conflicting assumptions is correct. Either way, I think we're better off.
 [00:46:53] You told me about this idea of an online repository for DAGs.
 [00:46:58] Right. So I think part of this ideal world might be something like this. Let's suppose
 [00:47:02] the American Cancer Society wants to put out a grant solicitation. They're going to give out
 [00:47:09] money to researchers. What would be great is if they had a structural DAG, a living DAG of whatever
 [00:47:16] cancer, they're going to try to research and say, here's the best explanation we have for
 [00:47:22] this system so far. We're interested in proposals that are going to address this corner of the
 [00:47:26] network, these nodes, these connections. Tell us in your proposal where you're going to wire in,
 [00:47:33] if you're proposing some new intervention or a new drug or doing something different,
 [00:47:37] how will that change the DAG? Tell us about the mechanism involved.
 [00:47:41] Hmm. That's a beautiful vision. I mean, we could do it in virtually any area.
 [00:47:47] Right. I think one of the objections to it might be, well, who's ACS to say how it works?
 [00:47:54] But that's the beauty of this system, too, is that if you are a heretic, if you're a dissenter,
 [00:48:00] you could easily fork the network and change it and say, here's my explanation. And then you're
 [00:48:05] now open to new ideas and people validating or invalidating your assumptions. So there doesn't
 [00:48:11] have to be a central authority. ACS could say, this is what we believe, this is the basis upon
 [00:48:16] which we'd like to give out research funds, but that doesn't mean you have to believe it. It just
 [00:48:20] means that whatever you do believe, you want to try to integrate into their explanation. And part
 [00:48:24] of your proposal might be, I think you're wrong, ACS, and I've got a proposal here that's going to
 [00:48:28] show you how and why. When you think about the broader causal community, people working on
 [00:48:34] on causal problems in different areas, in economics, health, epidemiology, business,
 [00:48:43] computer science. What would be your message to this broad causal community?
 [00:48:49] Well, I think my message would be look at the problems that these other disciplines are dealing
 [00:48:53] with, because you may see that we're actually all dealing with a lot of the same problems,
 [00:48:57] just wearing different costumes. We can talk about program evaluation in a public health setting,
 [00:49:02] And you might see that that's very similar to looking at an ROI calculation in a health insurance company or something.
 [00:49:09] Effectively, we've got the same problems.
 [00:49:11] We're just dressing them up differently as far as what their implications are.
 [00:49:15] But in their core, there's going to be similar solutions at work.
 [00:49:19] So learning from other disciplines, I think, is key.
 [00:49:22] And that can include looking at how they draw their DAGs, potentially, of course.
 [00:49:26] What are two books that changed your life?
 [00:49:29] Two books that changed my life.
 [00:49:32] One of them was probably statistical adjustment of data by W. Edward Stemming.
 [00:49:38] And this was just him sort of proselytizing about what does it mean to adjust data.
 [00:49:42] And it's a very straightforward, practical approach to looking at how to interpret things.
 [00:49:49] One of my favorite examples comes from early in the book, and he shows you a scatter plot.
 [00:49:52] He says, if you saw this, you would think it's just noise.
 [00:49:56] And it's just sort of dots up and down, no real pattern.
 [00:49:59] He says, but if we arrange them temporally, suddenly zoom, it's very clearly a line, right?
 [00:50:05] So his point is not look at things temporally.
 [00:50:07] It's just understand context and make sure that you're getting the right lens on whatever
 [00:50:12] data you're working with.
 [00:50:13] So that's one of them.
 [00:50:15] A second book that changed my life.
 [00:50:17] I don't know that I could come up with one as I sit here.
 [00:50:19] I'm sure there was one.
 [00:50:20] But in general, they tend to be the things that ask me to think differently about problems
 [00:50:25] and reframe what I think I know.
 [00:50:27] And so that's the approach I've tried to bring to everything that I do is, is there a different way to look at this problem? Is there a different goal we can infuse into our analysis? What do we know? What don't we know? And how can we use those things together to figure out something new?
 [00:50:40] If there was just one thing you could say to the new generation of researchers coming from machine learning, statistics, epidemiology, biostatistics, that are interested in doing science better, what would them be?
 [00:50:58] I think you would want to realize that the way you're taught how to do things in school is the only way that's tractable to teach it, but it's not the way to actually do it.
 [00:51:08] So, for example, you might take a statistics course, and they're going to hand you a data set for your homework assignment or your quiz or whatever.
 [00:51:16] They're going to say, here's these data, it's count data or it's continuous measurements of something, height, weight, whatever.
 [00:51:23] Here's distributions we expect it to follow. Assume all that's true. Now here's the model we want you to use.
 [00:51:28] And then they're going to step you through like a recipe. You know, look at the data in this way, check these assumptions, fit the model, here's your outputs, do your regression, whatever it may be.
 [00:51:38] check your diagnostics, and here's what it means.
 [00:51:40] Here's the boilerplate statement that tells you what an odds ratio is supposed to mean.
 [00:51:45] And that's well and good for teaching,
 [00:51:47] but when you get out in the real world, it's going to be more nuanced than that.
 [00:51:51] So you've got to start thinking about what is the model really trying to tell me?
 [00:51:55] What are the assumptions that are baked into that model?
 [00:51:58] And why am I using them here?
 [00:51:59] Or should I be using them here even?
 [00:52:02] Everything has assumptions and trade-offs.
 [00:52:04] So that's what you've got to be thinking about.
 [00:52:05] In light of what you know and what you don't know, what assumptions can you lean on and
 [00:52:10] which ones are going to hinder you?
 [00:52:12] Start thinking about the implications and the driving force behind everything rather
 [00:52:17] than just did I do the recipe correctly.
 [00:52:19] If you can do that, you'll be a much better scientist in my humble opinion.
 [00:52:23] This episode is a part of a series of episodes that we built together with UBC, the University
 [00:52:31] of British Columbia.
 [00:52:32] Yes.
 [00:52:33] causal ground, the causal cluster at UBC. What is your biggest hope regarding this cluster?
 [00:52:40] What would you hope to see as an outcome of this research cluster focused on causality
 [00:52:46] and graphical models in AI? Well, I suppose primarily it would be that there's greater
 [00:52:51] integration across the disciplines. As we mentioned earlier, there's a lot of different areas in which
 [00:52:56] people are thinking about causation. And it's not that some of them are right and some of them are
 [00:53:00] wrong. They're addressing different aspects of the same problem. So to the extent that we can
 [00:53:05] integrate them and again understand what are the assumptions and the limitations of each
 [00:53:09] and how might they complement each other, I think that would be a real great step forward.
 [00:53:14] Before we finish, what is one question that you would like to ask me?
 [00:53:18] One question that I would like to ask you. Well, you've built an audience of people who are
 [00:53:24] interested in causation. What do you think is their broad interest in this topic? Is it mostly
 [00:53:30] people looking to do computer science related large massive data estimation or is it people on
 [00:53:36] the bespoke epidemiological side like me or something in between? I think we have a very
 [00:53:42] mixed audience people coming from different areas so we have people interested in economics we have
 [00:53:49] some people interested in computer science epidemiology and health as well and the mission
 [00:53:56] of the podcast has always been to build a platform
 [00:54:00] where people from these different paths,
 [00:54:05] people with different experiences can come and
 [00:54:08] learn from each other. So when you said about
 [00:54:13] learning from people from other areas
 [00:54:17] because they are seeing problems maybe from a slightly different perspective,
 [00:54:21] that's very close to my heart and I think to our audience
 [00:54:25] as well. Great. Well, I'm glad to have been a part of this whole experience here because I think we're
 [00:54:30] talking to the same folks. Yes, hopefully. Robby, it was a great pleasure to have you.
 [00:54:36] Yeah, thank you. It's been fantastic. Appreciate it. Thank you.