Speaker 1 This is the evidence that he's using. I'm sorry. This is not at all comparable to counting asteroids. This is cherry-picking a bunch of astero uh uh experts Speaker 2 So once again it seems like you have a problem with Toby Ord and you should go talk to him or publish a piece bringing down his book or something, because this this sounds like a specific Yeah. Speaker 1 I have a problem with the methodology. And second, I have published pieces. And third, I did get um William McCaskill to take down his pieces because they're making the same mistake. Well Speaker 2 congratulations. I am I'm glad you managed to yeah, to uh make the discourse better. Speaker 1 Hello everyone. Um so this episode needs a little bit of setup, so I'm just doing a quick solo intro. Uh a couple weeks ago I was on the Bayesian Conspiracy Podcast with Eniash Brodsky and Steven Zuber. Um the episode just dropped on their channel yesterday. So I thought we had a pleasant conversation, but I learned after the fact that Enyash was not very happy with how it went down. So unhappy in fact that he published a blog post titled, I Have Lost My Ability to Be Charitable to Vaden, which I'll link to in the show notes below Uh in the blog post, he said I was gish-galloping and throwing out falsehoods faster than anyone can reasonably counter, and that he wishes he had this discussion with someone he could respect, and that he regrets having me on the show. So that's not very good. Um I wasn't terribly happy with how he edited edited it, but Anyash was kind um kindly gave me the unedited footage and permission to release on our feed. Uh which we're doing now so that viewers can decide for themselves. Um the only edit sorry, the only edit that I made was around the 15-minute mark upon Enyash's request uh to take out a false start to one of his comments. So uh you can read any ash's blog at deathisbad. substack. com, or you can check out the Bayesian Conspiracy at the Bayesian Conspiracy. com. And would love to come back if uh to clean up any of these falsehoods um if you guys would have me. But if not At least one third of us enjoyed the conversation and um hope uh listeners get something out of it. Speaker 2 Welcome back to the Basic Conspiracy. I am still Inyashbrotsky. Speaker 3 I remain Steven Zuber. Speaker 1 And I am still Vaydon Masrani. Speaker 2 Vaiden, welcome! Uh Steven and me are Bayesian Rationalists. We have a Bayesian Rationalist podcast. I discovered that you are a critical rationalist with a critical rationalist podcast. What the heck is a critical rationalist? Uh Speaker 1 critical rationalist, so it's kind of two things. Right now, it is both a online label that a lot of people on Twitter use. Annoyingly, um but once upon a time it referred to and it still does, and that's how I use the term, um the work of Karl Popper Uh so critical rationalism is is two label phrase to describe, say, his whole body's uh life's body of work. And I'm sure we'll go into it more, uh what those words actually mean, why it's called that, but um in general think of it as a placeholder for Karl Poffler's work. Speaker 2 Huh. Uh how is is it very different from Bayesian rationalism? Speaker 1 It's extremely different from Bayesian rationalism. Um so Popper spent a huge amount of time um criticizing um subjective interpretations of probability. Uh he does a foundational work in probability. He came up with new axiomatizations. He um so I don't know how much you guys know about him, but He was writing all throughout the the twentieth century, um working at the foundations of like quantum mechanics and uh logic and all sorts of different spots. Um but one of the main um differences is that He does not think that the probability calculus can produce knowledge, nor does he think that it makes sense to use the probability calculus to describe human belief or confidence in beliefs. Um and so it diverges quite wildly. I'm of course this is like a tip of the iceberg thing, so we'll be going into it much more. But um There's a lot of overlaps too, I should say. Um so there's a lot of similarities. For example, the idea that um you can never be absolutely dogmatically certain about something, you can always be wrong. Uh you should always be looking to get less wrong to eliminate error. Uh this idea that you can never be uh have perfect truth in your hand, i. e. in the probabilistic framework. You can never be perfectly certain with degree one. So he agrees with uh a lot of these principles. He just doesn't um use the probability calculus to um uh axiomatize or put on a firm foundation, uh, knowledge production. He says that knowledge is always conjectural, it's always open to revision, it's always open to error correction, but he does not try to formalize it mathematically as the Bayesians do. Speaker 2 Okay. I think real quickly, Popper, I recognize the name from back in the New Atheism days, he was the one with the falsification was his big thing, right? Speaker 1 Yep. one thousandth of his work. But that's what he's most popularly known for. Speaker 2 That's where I used him during the arguments. Speaker 1 Exactly. I Speaker 3 more of a the only uh oh this sounds really interesting. The only uh um concern I have is that we're not mathematicians. Uh so I think we might be a little out of our depth. Um Speaker 1 I'm not a mathematician either, but um I've I've done some math in my my PhD and stuff, but uh but none of the arguments require deep mathematics to understand. No. Speaker 2 So uh I guess we can on here because you have m may b you have issues with the Bayesians Speaker 1 I do. I'm in uh enemy territory at the moment. I'm loving it. No, territory, of course Yeah, we just favorite listeners. Yeah, and Yanash and I met um met at uh vibe camp a couple years ago, and then we kinda reconnected at a dinner party uh last week, and so all friends here, just just being uh cheeky. Speaker 2 So w what is it that you dislike about Bayesian rationalism? Because you you had like some strong opinions here. Um Speaker 1 I think it leads to errors. I think it leads to um formalization where that's unnecessary. I think it leads to For example, the idea that superintelligence is going to kill us all with some arbitrary probability. I think it leads to philosophies like long-termism, long-termism. Um and I think it leads to um fraud like Sam Bagman freed. Um to all of this. Um so there's many like practical problems with it. Um I'm kind of just doing a very broad summary at the moment. Um but even if none of those were the case, I would still have a problem with it because I just think it's false. I just don't think it describes how knowledge is produced. Um and I think as soon as one frees oneself from thinking uh about everything, probabilistically, um then one can make much more progress. Uh it changes the way that one interacts with other people and it changes the way one interfaces with uh with the world. So um this is a very broad survey, I know what you're asking. Tight little questions, so I'm trying to give tight little answers, but um but we can start diving into anything and specifically uh as as as you like. Yeah. Speaker 2 I'm kinda curious, so you think there's a hundred percent chance that AI won't kill anyone Or won't kill everyone rather? Speaker 1 I don't think it makes sense to talk about chance with regards to the future like this. Uh yes, so um this will be a maybe a bit of a longer answer Um but uh so for the listeners, um I I don't know if you had listened to the podcast I had sent um just uh in advance of this episode. Uh this was um uh sorry Speaker 2 Yeah, I listened to not all of it, but most of it. Speaker 3 We'll also put it in the show notes, but say the name of the case. Yeah, for sure. Speaker 1 The podcast is called Learning Bayesian Statistics. Um and it was episode 160. Um and so I think that's a good place if people want to get like a um more of a layman's explanation. I know that your audience is much more technical, so I'll just dive into the the the technical stuff. Um But there's two kinds of probabilities. Nice. So there's there's uh there's two kinds of probabilities, there's objective probabilities and there's subjective probabilities. So the example I used in that podcast was Toby Ord. And so Toby Ord in the precipice gives a number of probabilistic estimates of different things that could kill us, existential risks. So um for instance he talks about the probability of um say uh supervolcanoes, uh the probability of uh asteroids. Um When we are coming up with probabilities of these risks, what do we do? We look through the historical record, right? Um we uh or in the case of of asteroids, we scan the skies, we do like a grid, and we come up with some sort of uh rough and ready count. It's not gonna be perfect. but it's going to be counting something in the world. This is in the domain of objective probability because there is an object to study, i. e. something that is independent of the subject. And most specifically there's in all cases a dataset associated with it. So other examples of objective probabilities would be things like um what's the probability of having a miscarriage in your first trimester? Uh so you can look at all of the various um pregnancy rates over the last A couple decades, uh, you can make some assumptions about what you consider outliers, and then you can count, and you can divide by a total. Um how can you do that when we talk about the unbounded future? When we talk about the unbounded future, we are now switching to a subjective form of probability. um counting some outcome of interest and dividing by a total. It is assigning a credence, uh i. e. some scalar value to a belief that you have. Now, uh I'm not claiming that you can make no predictions or no probabilistic estimates about the future because meteorology does that. Right? So every time you have a weather forecast, you're making some estimate about the future. Um but this is heavily constrained by data. And it peters off after about like 14 days. It's very difficult to to go that much further. When we talk about what is the probability of P doom, what are we counting and what are we dividing by? So direct question to you guys. Speaker 2 Uh I don't give AP Doom because I don't I don't think you can put like a specific number on it like that. I just think that it is a thing that could happen and I'm worried about it and it's got a decent chance of having it Speaker 1 friends But that's very different than asking for what is a chance. A chance is a number, right? So if you say to me, like, um, what is like what's your hunch, man, that like AGI is gonna happen and how bad is it gonna go? I'd say I'd say it's probably pretty unlikely. Yeah, uh and so we could talk about like speculations about the future. But the problem when we start saying the probability is sixty-three percent or the chance is fourteen percent As soon as we add a number to it, now this gives us license, as Toby Ort did, to compare numbers that have been made up against counts of the real world. So Toby Ord's whole point in the precipice was to say ignore all of the supervolcano risks, ignore all of the um uh asteroids, ignore all of that because the chance is so small Compared to the chance of superintelligence, which is I think he said one in ten and then he aggregated everything to be one in six. Uh hold on a second. You're comparing things that can't be compared as soon as you make that uh mistake by comparing subjective probabilities with objective probabilities. And That can be avoided by just not using numbers. I'm just saying, I think it's pretty unlikely. I think it's really likely. And that's fine. And we can reason about the future this way. But we can preemptively avoid a serious mistake by simply saying it doesn't make sense to talk about the uh expectation about the future, sorry, in uh like a expected value sense, or chances about the future when we're talking about unbounded futures. If we want to talk about something specific like what's the chance that you're gonna have uh examples I've used in the past like um uh an influx of hospitalization rates uh during Christmas due to drunk driving. Um Okay, we can talk about the feature in that sense because it's a limited, constrained environment and we can roughly count and track stuff. So um when you asked me what you thought my chance was about uh superintelligence and AGI, I just reject the question. I don't think we should or can put Numbers on it, but that doesn't mean we can't talk about it, doesn't mean we can't reason about it, doesn't we can't think about it, it's just we have to be careful not to make that particular class a mistake. Speaker 2 Well I mean I was just asking if you thought that it was zero Uh Speaker 1 again, you're uh I'll say I will use soft squishy language. Um and so I don't think so. Speaker 2 Okay, yeah. That's I mean that is that is what I was asking. Speaker 1 Yeah, sure. Speaker 2 Okay, okay. Oh, okay. Uh I I don't know. I don't I don't think this is like a thing to get that deeply upset about because on the one hand it's nice to have numbers because language can be very uh Very subjective. Uh you can say I don't think it's very likely, and if someone says I don't think it's very likely, like there have been many surveys taken about this. People say unlikely is anything from 1% to 70% Which to me is like first of all, if I thought something was unlikely I would put it to under fifty percent. At the very minimum it'd be forty nine percent. But uh people mean different things, I guess. Uh so I I would like to get some sense like when my doctor says, yeah, it's unlikely that you have cancer, I'm like, okay, so unlikely like 30% chance? Cause then I want to do more tests, or unlikely like one in a million and I'm just being a paranoid person And that is like I don't need to have an exact number, but I would like to have a ballpark if he means like, you know Yeah. And that that is the kind of thing that I think people want when they ask for numbers? It's like, so do you think there's like a one in a million chance and we should worry more about the uh the asteroids and the super volcanoes? Or do you think that's like as bad as our chance was of having nuclear war during the eighties and actually we should put some resources into avoiding this? Speaker 1 People often want to ask for how one got one's numbers, not just the numbers. So when when you talk about so I'm gonna continuously enforce and suggest that we keep separate objective probabilities with subjective probabilities. So you conflated the two in your in your example here. So when you go to the doctor And you ask the doctor, What's the chance of me having cancer? And he says it's very uh unlikely. And you say, Why? Right? And then he says, Well, because the particular kind of melanoma you have Um I've seen it, uh we can here's a bunch of studies. They tracked 100,000 people who all had similar uh kinds of melanomas. You fit this particular uh demographic, so we're gonna condition on your height, your weight, your age. Uh that's gonna give us a smaller uh denominator. And that of this small set, now we know that there's 13,000 people out of uh hundred thousand people who haze developed cancer based on the evidence that I'm seeing, right? Um that's great. And so by all means, map a likely to a methodology Don't map it to a number arbitrarily. And so in your example, you are making the mistake that I'm saying is rampant within the Bayesian rationality community, which is not distinguishing between chances of arbitrary things in the future against things that you can count, you can measure, you can experiment, and you can look at an Excel sheet with a dataset. Speaker 2 I would rather have that information as a subjective number than him saying, Well, I can't tell you because there's no objective numbers behind it. Like that's not a good idea. Putting uh I mean uh if you're saying that Speaker 1 the other option is you lay it out experimentally. Speaker 3 Not to give you their vibes based answer of like, you know, I had a good lunch and it's it's Friday I'm looking forward to the weekend. I think you're probably good, man. You go, you know I I think one one Speaker 2 if that's his reasoning I'm firing my doctor, yeah. Speaker 3 But you do but you said you don't care you you don't want his his methodology, you you want his his number, and it's like I mean I I want his num in the case of a doctor, it's like you want the number based on The everything that Baden laid out actually. Speaker 2 I'll grant you I would like to know why he has come to that conclusion. But like I can't double check his numbers. I'm not a doctor guy. Okay, sure. In theory, I can. Uh in practice. Speaker 1 Yeah. In practice. You can do it in the call. And also additionally, additionally If he gives you a subjective credence and it disagrees with what an experimental result is gonna say, what are you gonna use? The experimental result. Speaker 2 The experimental result. Because the credence Speaker 1 doesn't really matter. Yeah. Because you care about the methodology Speaker 2 Yes, okay. But again, if someone were to tell me I cannot uh tell you what the chances are of go of the Soviets launching their nukes and starting a a nuclear war Because we don't have, you know, any base rates for that, we don't have a methodology for it. I'm I would still want to know, like, telling me, I don't know maybe, is a bigger mistake than telling me like Almost certainly not one in million chance or actually yeah this is pretty likely. I would I really would like to uh have some idea of the order of magnitude at least. And I think uh saying like well we can't say because there's no objective numbers is worse. Were you not saying that Toby Ord comparing uh his one in ten chance of AI killing everyone to a one in hundred thousand chance of asteroid strike is bad and he shouldn't have done that? Speaker 1 Yes. But that's not the same as saying that we can't make reasoned judgments about what's going to happen. Speaker 2 Right. Well I think if you looked into his methodology of why he got the one in ten, you could argue with that, but I think honestly it would be good to know if uh If it is a reasonable estimate to be one in ten or one in a hundred or even one in a thousand, uh I would I do think that should be prioritized over Nasteroid strike, because nasteroid strike is much less important in that case. Even though the asteroid strike is objective and the other one is subjective. Like you can look into the subjective numbers and be like, yeah, I think that seems more or less right, or like that's really way off. Um so we Speaker 1 talked about to be ordered but let me go back to the previous example you used about um the Soviet nuclear strike. Um so I don't know, you and I are sitting in uh Reagan's office um and uh Reagan's like, hey guys, what's your Uh what's the chance this is gonna happen? I would say, I don't know, but let's talk to our spies who are there. Um let's run some experiments. Uh so the claim I'm making is not um If you get rid of subjective estimates, you just have a nihilistic void of no information forever It's we need to use other approaches to answer this question. Let's get some people on the ground. Let's l er do aerial photography to see if there's um I don't know any movement in in uh Moscow that indicates that they're starting to big Speaker 2 I mean if that's your argument, then I don't think we actually differ. I think we would Speaker 1 agree with you one. It gives you a false sense of knowledge when in fact you should be like, we can't really assign a probability here, so let's figure it out some other way. And I can give you other ways to figure stuff out and we can talk about uh what Toby Ord actually in fact did. What he actually in fact did was wrote a whole book of arguments. Um that were valid and we can think about those arguments, we can decide if they're reasonable, we can argue against them. And then one in two pages he puts a little cute number on it, which is completely Bullshit and made up as he says in the book, and he says that what he's gonna do is he's gonna start with just a fifty percent chance and then reduce it arbitrarily based on his vibes. Um the mistake he makes is um Comparing two methodologies that should not be compared, and then other people who don't have the time to go into subjective nature of probability and the problems with it, then use that rough and ready heuristic to allocate their donations to different organizations. This is the mistake. I'm saying don't uh and we can talk about uh what William McCaskill did in his um uh piece, which he took down after my critique of it, called a case against or my critique was a case against long-termism his paper was the case for strong long termerism, he used the exact same um duplicitous reasoning in order to tell everyone to stop donat donating money to um anti -malarial uh organizations in Africa and start giving it to graduate students on laptops who want to write blog posts about how scary AGI is. And so this resource allocation thing, which is only done via this false comparison between numbers is the thing I'm critiquing, is the thing I think it should not be done. That is not the same thing at all as saying we cannot reason about the future, we can't argue about the future, we can't run experiments to try to make more progress I'm just saying quit making up fake numbers in your head and thinking this gives you any knowledge whatsoever about the real world. No, it doesn't. You have to talk to people, you have to go experiment, you have to go look at it, you have to go run um tests. You have to produce arguments and have people argue against them. And none of this is nihilism or just we can't do anything, man. It's it's let's just stop doing the one thing that we know is leading us to error and let's start doing stuff that works. Speaker 2 It seems like just a complete straw man though because I I mean sure you bring up Toby Ord's book where he uses that as an example maybe and then uses his own number, but like Every rationalist that I know is constantly going out trying to get more data, trying to compare their uh models against reality, trying to argue with other people. Uh how can you do that? I've seen a lot I uh I I don't know, let's get into long termism in a second, but uh just in general. In general the the methods I've seen. They uh they publish how they got their numbers, they publish what they think the things are. The in most cases they'll actually publish like a little online model that says like, here's where I put my numbers, here's what I think they are But please adjust this at your own leisure. Like, here's where I put my slider for how effective I think a malaria net is. If you have different data that you think different, please adjust the slider and, you know, make your decisions accordingly. And I mean, I think that's great. That isn't that what we want? To encourage people to argue and to look for more data? Speaker 1 No, that's not what we want. I think Bentham's bulldog is a perfect example of why you don't want this, because you can use this technique to reason yourself into any wacky position you want whatsoever. He produced a bunch of numbers and he proved that Christ or that uh God is real. Or he proved with high probability. Um and he gave you three hours of just confusing math that made no sense to anybody and he could reason himself into any arbitrary conclusion he wants. I think the one of the other things he's talking about now is that we should care much more about insect sentience sentience than human being sentience. If he was to participate in the war against malaria, he would choose the side of the mosquito because of this kind of reasoning approach. Um one can reason oneself into any possible conclusion they like, and there are better methods of reasoning that don't fall into this trap. Secondly, you talk about everybody you know does this. Yeah, that's because this community in particular does this. I'm in multiple communities. Other communities don't reason this way. For example the AI research community at UBC or at Stanford. No researcher is doing this. It is one community that does this. Which is the effective altruist Bay Area Rationalist Community, because they all read one book, which comes from Elier Zukowski, and he read one book, which comes from E. T. Jaynes. So we could trace the lineage about why this community does this Um but the fact that you say everyone in your community does this, thus it's a good thing to do, um is just because you haven't stepped out of this community and checked out other communities and seen how other communities reason. Um most communities don't in fact reason this way, and most communities are able to make decisions just fine. Speaker 3 I I don't want to exactly backpedal, I just want to like kind of Speaker 1 lay lay the Speaker 3 Yeah, well and I I didn't uh I didn't put my flag down at the beginning. Um like I think Yeah, like what's my P Doom for example? I don't give a number either for similar reasons. I I feel like it's pretty vibes-based. And like that that said, like for the purposes of a conversation I might give a number like 10% or something because I I I want the person I'm talking to to be able to understand what I mean by maybe, right? Um and so like Yeah, no, I that's fair, because like what what does 10% mean if I'm talking about a thing that might happen in the future? I I get ya. But it's it's more just like I think it's a way to help kind of anchor feeling and motivation because if I say, well maybe, and that could be like, do they mean maybe when I say that maybe this is a winning lottery ticket or maybe, you know, the sun will come up tomorrow? Um Like they w if I if I don't give anything to ground it on, I feel like my anyone I'm talking to would be um avail like it wouldn't be unreasonable for them to to think Oh okay, maybe. But he's he's Stephen's a scientist. He uses maybe to qualify everything. Stephen's not really a scientist, they might just think that. Um and you know, Steve see I might say that the sun has a high chance of coming up tomorrow, but we we haven't been to tomorrow yet, so we're not sure. So if he's if he's talking like that, then maybe means nothing. Or uh it's um yeah, d I I guess that's part of it. The other thing that like I you know, I'm I I guess yeah I'm not I'm not well poised for like a uh Bayesian versus the opposition kinds of argument because I'm not I don't know. Um not immersed in the mathematics of it. It's more it's more the mindset. And I'm wondering where your framework comes in on this. And we can table this one for later if you want. But like For me the the big value of this framework of thinking is like my beliefs aren't on off switches, they're probabilistic. Um or if I don't want to say probabilistic I can at least say that they're gradient, maybe. Um I have I have degrees of confidence in things rather than running around being sure that this is either true or not true all the time. Uh and I find that hugely beneficial. Mm-hmm. Speaker 1 Yeah, there's um there's a flavor of Bayesianism. that I have less problems with, uh and that's I think best represented by Julia Gailiff. Um uh largely I take the so when she talks about Bayes Largely the message she's trying to convey, and please correct me if you if you disagree, is like, hey guys, like have an open mind. Like don't be extremely certain about one thing or extremely f um uncer uh certain about the negation of that thing. Just like be open and available to disconfirming evidence. Don't be too dogmatic. Be more of a scout than a soldier Um that's all great. That's all fine. Uh to the extent that Bayes is um inculcating that into um into people who learn about it. This is kind of how like Stephen Pinker talks about it. Um that's all fine. I think you can get there via critical rationalism in a better way, but uh to the extent that that's all people mean when they say Bayes. I don't really have a big problem with it. I have much more of a problem with it when it's used for ideological purposes to funnel money away from say development programs in Africa towards one cause in particular, AI risk, AI safety. And that kind of argumentation I think wouldn't get off the ground ha if it didn't use a lot of, let's say, probabilistic smoke screens. Um in the same way that you guys spent three hours trying to combat And those bulldog, this is what um P Doom does for people who are just getting into like the AI safety debate. On your question or on your comment about like um If someone asks you what you think is going to happen with AGI, and you're like, um, I want to say it's 10%, but really I just want to say it's kind of like unlikely. I would say I think it's unlikely and then here are the reasons why. And just immediately pivot to start talking about your arguments and the things that you've found plausible, the things you found implausible. Talking about relative comparisons, like the 10% number, um all that means is uh so you when you talk about Bayesian credences and stuff, there's no frequencies, there's no divisions, it doesn't really map onto anything objective in the world, it just maps on to relative things that you also feel that confident about. So you can just be like, so for example, what you said like uh I'm not 99 % certain. What does 99% mean? It means like Um I'm not as certain about this as I am that the sun will rise tomorrow. So you're just giving a relative comparison between this proposition and some other proposition. Um but who cares? I don't care how you think about this relative to what the sun's gonna rise tomorrow, I care what you've heard that makes you worried about AGI superintelligence or what you haven't And let's argue about the actual substance of the conversation. Um likely, unlikely, like the this doesn't mean much when we're talking about the unbounded future. Um and in particular it focuses way too much attention on your beliefs and way less attention on like Okay, what can we conclusively say we know? What can we conclusively say we don't know? Why? How can we figure out stuff that we don't know? And this is an objective view of knowledge and an objective view of science. Um we can figure this out rather than ah I gotta spend a lot of time calibrating and f and tuning my beliefs so that they're perfectly matched to reality. Well, by the time you've done that reality's changed. And so the more time you spend thinking about your beliefs uh the less time you're spending understanding the real world. And so that's another reason why I think that the focus on subjective beliefs is unhelpful. Speaker 2 You're gonna go out on a limb? Oh go ahead, Tim. Speaker 3 I I I was gonna say I I'm finding myself agreeing with a lot of what you're saying. Like I think Certainly Topio 's saying let's start at fifty and then just throw my vibes at it fifty percent I mean seems kind of nuts for whatever claim it actually is that he's measuring. I think it was AGI in the example you gave. Like, but to say let's start at fifty, that's like saying, well there's two outcomes, either it goes bad or it doesn't go bad. So that there's two chances let's say fifty percent. And that's that's not how odds work. Uh wasn't that a dumb and dumber joke? Speaker 1 So the fifty percent thing is so important to flag, and if your listeners are infuriated with me up till now, just listen to this one point because I think it's very important. Um Uh if I say that the outcome of m um um that there's a fifty percent chance that I'm gonna get into a car crash on the way home Speaker 2 Oh shit. Don't drive home. Speaker 1 Yeah, exactly. Fifty percent chance. That means that you're taking an objective view of like uh one and two times or fifty times out of a hundred, you dead, right? Um if you take a subjective view, what fifty percent means is maximum ignorance So 50% from the subject of Bayesian means I have no information whatsoever to adjudicate between outcome A or outcome B. So I need to be maximally dumb. This is a state of pure n lack of knowledge, pure utter ignorance. If you had any information to tip you that the coin is going to flip more heads than tails then you would have knowledge. You'd have some less ignorance. Um so Toby Ord is starting from a place of maximum ignorance And he's comparing this to what astronomers and geologists have spent a hundred years measuring and counting, and there's absolutely no ignorance on the side of the astr uh astronomers um and the uh geologists, but fifty percent much bigger than one over one one thousand right? Fifty percent much worrying much more worrying than one over ten to the nine. Um this is a mistake. This is a mistake because he's conflating between two notions of probability and he is intentionally not communicating this to his audience. This is a huge problem, it's a huge mistake, and it leads people to allocate resources differently than they would had they not been tricked in this fashion. Speaker 2 Okay. I have well, okay two things. The first one I want to touch on very quickly in that uh the whole fifty percent thing, it's a formalization of where to start and obviously does not track to how people use that in day-to-day life. There are actually Speaker 1 comparisons. Right. Fifty percent is greater than one on one trillion. Speaker 2 Yeah, yeah. Uh but maximum Speaker 1 ignorance, maximum less ignorance. Speaker 2 I understand. Th this is a formalization. There's rules on how you actually apply the calculations in order to get to some level of knowledge. But We're setting all that aside because first of all we don't have, you know, twenty hours to get into it, uh into how to do all that. And second of all because I don't think that's actually the uh complaint at issue here. Um A quick aside, uh you say that like Toby Toby Ord's numbers are bullshit and that's fine. Uh I haven't looked into his numbers, I don't particularly care about his numbers. Uh the I think the fact that pretty much every expert in the field puts some single digit, at least single digit percentage, uh of worry about this, many double digit percentages, to the point where if you take an average of them you get eighteen percent. Like If we're actually asking experts and looking at their methodology and where they came up with those numbers, I think that is something we can actually go on. But let's put all that aside too and say like, you know what, we're not talking about the AI question and what the experts think right now and how they came to their numbers, because again that would be a multi Project and there are many other podcasts that have done that. I want to go out on a branch here and cut to what I think is the root of the issue. When you described, you know, the Bayesianism you don't have a problem with, you described basically Bayesian rationalism. You described like how everybody talks and does this, and I um And when we were like arguing just now, basically we came to the agreement on everything of substance. I think the main issue not issue. I think what is uh driving this is that you find that the AI The risk of AI existential threat is vastly overblown and you're upset that money is being diverted from other causes to this and uh And you want that to stop. And I'm you know, I can see that point of view and I'm happy to argue about that, but like that doesn't feel like to me it's an argument about Bayesianism, I feel that it is an argument specifically about AI risk. And um I guess that's okay, but I don't want to get straw mining of Bayesianism pulled into this if that's not the actual bottom line. Speaker 3 Is that a quick question before you respond to that, Baden? Was was that intuition there, Inyash, based on like conversation you guys had last week when you guys said that Speaker 2 when we were talking last week it was entirely about like why shouldn't you put numbers on things in order to clarify edge cases And uh I this is just the impression that I got just now due to the uh talk about long-termism specifically and money being ful funneled away because people are being tricked. And Speaker 1 is that the impression you got from my comments? Is that the impression you're doing? Speaker 3 No. Um I th I think that like I think he pointed at areas where this kind of reasoning and I think it's a Toby Ord like explicitly intentionally didn't give his audience like the the correct picture. I forget exactly your phrasing, and I haven't read his his uh book. Um but like to me it sounds like he made a blunder. Speaker 1 What was that? Speaker 3 Oh thank you. Yes. Um it it sounds like he's just he's he made a blunder that uh Well, I tell you what, I didn't mean to to take too much of our time. For yeah, one data point, I didn't get that the same takeaway that Enoch had. I hear what you have to say. Speaker 1 Yeah, did not represent my argument. Well that uh maybe I was just mis mis um uh stating it or or not stating as clearly as I should. Um Yeah, I have independent reasons to be um against Dumerism. Uh we can shelf all of that. That was just an example of how I think this reasoning can get um uh lead people astray. Um an additional example would be Sam Bankman Fried, and additional example would be long-termism, and additional example would be William McCaskill. These are all different examples that stem from the same uh mistake. No, it is not the case that Julia Galiff um is the same as all Bayesian rationality, and let me just give you some quotes. So um the thing that I'm rejecting is this idea that the human brain and the human reasoner, the rationalist must put credences on their beliefs. That it Gaeliff doesn't say they must. Elizukowski does. So let me just give some uh quotes here. So this is from one of his blog posts, Blu A Beautiful Probability. Old school statisticians thought in terms of tools, tricks to throw at particular problems. Bayesian at least this Bayesian though, I'm not only speaking for myself. We think in terms of laws. Bayes' theorems are elegant, coherent, optimal, and provably unique because they're laws. The laws of probability are laws, not suggestions, but often the true law, capital L, is too difficult for us humans to compute. Um To Bayesians, the brain is an engine of accuracy. It processes and concentrates entangled evidence into a map that reflects the territory. The principles of rationality are laws in the same sense as the second law of thermodynamics. Remember this when you plead to be excused just this once. We can't excuse you. It isn't up to us. From the blog post, no one can exempt you from rationality's laws. This is what I'm rejecting. This is what I'm objecting to. This idea that the only way to be rational, the only thing that one can do, one must do, is force the probability calculus to uh apply to every single belief one has about every single thing. And that every thinking that every everything you think And how you reason must be filtered through this one lens, this one framework. Even if all of the examples that I gave, none of them happened, I would still reject this because one, I think it's false Two, I think it is premised on Cox's theorem, which has three assumptions, none of which I think really map onto reason reason or rationality. And three, because Karl Popper's theory of conjectures and refutations this gives a much nicer and more elegant um description of how knowledge is produced and how one should in fact reason that preserves all of the good stuff from Gaeliff. um without trapping oneself into um thinking that credences are like a neurotransmitter or something like something you have in your brain that you have to do. You do not have to. This is an assumption. And as soon as one frees oneself of this assumption, one F just it's like a window opens up and there's just a brush of fresh air a f breath of fresh air blows through the room. Because you don't need to s do integrals when you're trying to decide which school to go to or who to love or whether or not God exists. You don't need to marginalize, you don't need to compute conditional probability Distributions, you don't need to compute the posterior. This is what Bentham's bulldog has spent a whole career doing, and he just leads himself to insane conclusion after insane conclusion after insane conclusion. Um There's no guarantee that you will be more rational if you think this way, and in fact, there's a whole class of mistakes that you would never reach to uh um get to, such as prioritizing insects or computer game characters over human beings um if one didn't make this kind of mistake. So no, it is not just my um Dislike for the AGI conversation, although we can talk about that too. But that conversation cannot be had without going into the philosophy that undergirds it. And so the more you argue about this, the more you start start hearing about Dutch books and the more you'll start hearing about credences, and then you gotta go right to the base. And so that's what I've done because I've been forced to, but uh I I reject the base, not the conclusions, although I reject the conclusions too. I've Speaker 3 can I get a clarifying um I mean this is not like a uh a annoying philosophical question of Uh trying to to pin you down to a to a silly answer. You used the word knowledge Um I think there's at least two different kinds of things we're talking about. You know, like um I'm taking a vacation next week in the tropics. And I uh do I know it's gonna be warmer than it is here? Like is that knowledge like I g uh without checking the forecast. Um I well let me think of a better example. I guess there's so there's knowledge of like Speaker 2 that's a great example Do you have a good reason to believe that it will be, in general, warmer in the tropics than it is in Colorado? Speaker 3 Right, but like since I could since I can just check that it makes it a less easy or it makes it a less useful example. But like Speaker 2 a belief and you have some credence in that belief. Speaker 3 Sure. But as opposed to like the knowledge that I have that this thing has about a third of you know the volume of water left in it. Like that that's something that I can look at and that that's that's knowledge. Speaker 1 I think he has a belief about it. I reject the assumption that it necessarily must be mapped onto a credence. Speaker 2 Do you think that he doesn't have a certain credence? Speaker 1 That's the point of my last burst, that the credences are an assumption. Um it's a I don't have a credence about stuff. I have beliefs about things. But credences are not something that everybody just has. It's an assumption. It is not baked into the laws of physics. It's not part of the brain. You can't cut a person open and find a credence like you could find like the dopamine molecule. A credence is a number between zero and one that is associated with every proposition, every belief that is supposed to, according to the Bayesians, um track your subjective feeling of certainty. Such that when you become maximally certain, the scalar number goes to one. And when you become maximally uncertain, the scalar number goes to zero. I don't have numbers associated with my beliefs Um some beliefs I can, like for example, we talked about miscarriage rates and stuff, and those numbers come from uh scientific methodology. Um but for uh Question like um what is the uh number of people who are gonna wear Nike sneakers in the year 23 -34? I don't have a number. Um I Speaker 2 think your assertion that uh everybody has a number is just wrong. That's not what Bayesianism says. Like Speaker 1 No, but that's not what I said. I didn't say that everyone has a number is wrong. I'm saying that the assumption that everyone must have a number is an assumption, not like a law as well. Speaker 2 Nobody in Bayesianism thinks that everybody has a number for every belief. That's a good idea. Speaker 1 That's a definitionally what a credence is. Speaker 2 Right, right, right. But like you don't have to do it. Speaker 1 If you say credence, then you mean that Speaker 2 Yeah, but you don't have an actual number on your beliefs. Like Stephen, you don't have an actual number on how likely you think it is that it's warmer down there, right? Speaker 1 So if you don't have a credence to this belief, then everything that Yukowski said in his blog is wrong. Speaker 3 W I think that's worth unpacking. 'Cause like yeah, uh I yeah, uh go Lo and I ca I I Introduction a s second ago when you know I said like there you know you don't there's uh not everyone has credences that it's like you can just read the quote again. 'Cause it sounds like Ilikowski is saying that Uh if you're a rationalist you you darned better have one. Yep Speaker 1 and uh you can read all the quotes again if you like, but um yeah This is what he means by laws. To Bayesians, the brain is an engine of accuracy. It processes and concentrates entangled evidence into a map that reflects the territory. The principles of rationality are laws in the same sense as the second law of thermodynamics. Remember this when you plead to be excused just this once, we can't excuse you. It isn't up to us. If you say the word credence You are assuming that it makes sense to map a single scalar value to every possible proposition that you can think, such that that scalar tracks your certainty. If you say, no man, I don't think that you should do that with every possible belief. There's some beliefs I don't have credences for. Then you are in a direct contradiction with Yukkowski, who says that it's a law. It's not a choice. Um Speaker 2 that puts Yatkowski in direct contradiction with the Yudkowski, because I mean I've talked to him and seen this. I'm sure there's video of him saying this, but sometimes you can just ask him a question of like, how likely do you think that is? And he's like, eh, I don't know, somewhere between thirty and fifty percent Like this is just a thing humans do, and uh it by you claiming that he claims everyone has some specific number for every single belief, is just objectively false Speaker 1 Dutch book, a whole Dutch book is an argument that you must assign credences to your beliefs, else you could get into a gambling scenario such that someone could take advantage of you The argument Dutch books are an argument that one must assign credences to one's beliefs. I can tell you the problem with Dutch books later if you want, but um that's the point of the argument, and that's what a credence means, and that's what Bayesian epistemology says and uh edited listener can just fact check everything I'm saying. This is all true. Speaker 2 Real quick, before we continue, did did we just lose my video? Speaker 1 Yeah, sorry. Speaker 3 Yes, you're frozen. Speaker 1 Oh you're frozen Sometimes it comes back. Yeah, turn it off. Also, since we're paused. How are we doing fellas? I'm I'm liking the heat. I hope there's no bad vibes. I'm having fun. Yeah. Yeah. Yeah, 100%. Yeah, for sure. Because I didn't get back to your knowledge question, so we can talk about that too. Speaker 3 Oh yeah, we'll get we'll get there too. But also like Uh not throwing shade at Inyash but I didn't like how he's like, well I think it's just this this is why you're upset. I was like, he didn't say that yet, man. Like let's let's let's hear where he's actually coming from. But like it's it's also fun to argue. Speaker 1 I love the sport, so Inyash, I'm loving the tone, I'm loving the intensity. Don't soften it even one iota. Keep it keep it up. Keep it up. No bad vibes. Uh I'm I can Get us going in if we want. I know there's a bit of technical difficulties, but by all means, Inyash, if you had a question or want to challenge me on something, then that might be a good idea. Speaker 2 If the the general claim is that the brain does this on a subconscious level, uh like that is not uh That is not we don't specifically have number, as we were just saying, for most of our beliefs. Not even Eliezer has a number for his beliefs. Uh the Bayesian claim that the brain that does this Uh may very well be true. I would again refer you to not again. I would for the first time refer you to the predictive processing model of the human brain. Oh, okay, well. Yeah. Speaker 1 Yeah. Speaker 3 Okay. And I I'm curious what uh you know all right, so there's a couple things 'cause we we do have we don't have a hard stop anywhere, but we do have like time constraints vaguely in our lives. Um I wanna make sure we're actually hitting like the crux of interesting disagreement and not getting s you know, tied down on like uh one snippet of one sequence post. Um I think like the th the br the broad claim is like human brains kind of work in a Bayesian way And I think that does map to what I underst like how I imagine it working. You know, our our fuzzier, dumber ancestors would be like, can't go over there, maybe maybe lions Well we're hungry, that is outweighing, you know, the maybe part of lions over there. Um and like it it's all, you know, sub sublingual sub uh th they're they aren't computing numbers, but they are comparing like their intuitions that are informed, you know, to the uh they're probabilistically informed intuitions, right? Uh even if they're not throwing numbers at them because they can't do math because they're basically monkeys. Um I I still think like something Bayesian-ish is happening in there, right? Speaker 1 There's like learning. So you have learned throughout life that when there's a rustle in the bushes Um sometimes that Russell could be a predator. And so you've seen this a few times. Uh then you have a little conjecture about like uh oh big animals like to hide in the bushes because they like to jump out at me. Um and so then you adjust your behavior accordingly. Um I can give you that story. I don't have to talk in terms of probabilistic language or the the prob probability associated with that. And arguably many things we do uh we can't come up with a probability, but we still have to act anyway. What's the probability that the Wright brothers were able to successfully make an airplane? Well, how can you possibly count that probability because there were no airplanes or And what's your reference class? It's impossible. But they did it anyway. They did it because they had ideas about how to fly. They had a uh bold sense of optimism. And this is how all scientific invention and discovery works, not because there's some probabilistic guide, because the vast, vast majority of experiences you have throughout the day are singular and unique. Some of them are different, of course, or some of them uh uh are repeated, um but even a repeated instance like me going to my office today to record a podcast is only repeated because I'm selecting that which counts as a repetition. Like uh today I have um I don't know my earbuds over here. I didn't have it last time. Is that a repetition? Is that not a repetition? Well I'm just Cherry picking, what counts. So this idea that you can only learn or act or be in the world because you have some probabilistic estimate that has been guided by a whole bunch of repeated occurrences in the past. um just doesn't map out to a world which is constantly changing and dynamic for which we often see things for the first time. But we still do have to act. So so I would claim that um Uh Speaker 2 This just seems like a complete mischaracterization of Bayesianism. Like this is uh this is constantly a thing that is said that is oh oh I'm sorry. But no this is just a thing that is said, though like we m all all must reason under uncertainty. And the d there you still have an expectation that when you go to your uh podcast studio it's going to record your voice and that there's going to be me on the other end or whoever you are planning to talk with. Like you have expectations about every single thing in life, despite the fact that we all live under uncertainty. Uh and it seems like you're d very upset about specific numbers being used. Well okay, okay. Maybe not upset. Um the thing you dislike about basicism. Okay, okay, sorry. Alright. Speaker 1 Yeah. Speaker 2 The thing you dislike about Bayesianism is that what, it tries to talk about this in a formal way. Speaker 1 I mean I and I Speaker 2 can not that the Tush talk about it in a formal way, but that's the same thing. Some people use it in a crazy way, like Speaker 1 Could you could you repeat back some of the things I've said already about, for example, the problem with um comparing fifty percent probability against asteroid strikes? Because I feel like I've said it a few times and Steven gets it. But when you say it back to me, you're not perfectly capturing my argument. So Speaker 2 Well, I mean the thing that when you're saying things like uh Bentham Bulldog's reasoning makes him value insects more than humans Uh basically everybody agrees. Focus on the Toby Orthe's crazy Speaker 1 Toby Ort example. The Toby Order is the cleanest one. Yeah, yeah. Speaker 2 He thinks that there's a one-in-ten chance that AI is going to destroy the world in the next century. And that that that uh absolutely overwhelms the risk that we're gonna be killed by an asteroid in the next century. Speaker 1 No, it's not that he assigns a lot of things. To make up a number and compare it to experimental results. Do you agree that was a good thing to do? Let me give a fresh different example. Um just to clean the slate and let's try it from a d different perspective. Um so during COVID, uh one of the research projects that I was on was to try to model the spread of of uh COVID nineteen through um a county, a county in like Pennsylvania or something. Um and so we had to come up with a probability of the future. We had to come up with a probability of like what is the chance that I think one of the things we ran um That greater than ten percent of the population would be hospitalized to due to COVID nineteen. Okay? Speaker 2 Yeah. Speaker 1 Um reasonable question I'm not saying, hey, don't use probabilities here. You can't no, of course, this is how you compute the R naught value. This is a very scientifically rigorous uh thing. So how do we do it? Okay, so What we did was we took a giant C plus plus simulator, um so think like a hundred thousand lines of code and eat and it was a agent-based model, i. e. There is a little probability that each citizen is gonna get up and go to work and touch some bacterial agent and what's the probability that the male like the um the um the postal service is closed like a whole town like think think of the sims actually the sims is a good mental visualization so you have a full like the sims all the people right um and Baked into the simulator is like fifty years of epidemiological knowledge about how things spread, how this town works. Um and then when we come up with a probability We can tweak the parameters at the top and then we can run simulations. So we can run like um uh like it's called Monte Carlo estimates, you just run a whole bunch of simulations. And then you tweak some of the parameters and you say, okay, um if the R naught value is this, then we will expect to see this percentage of um people uh uh um hospitalized due to COVID nineteen. This is great. I'm no problem with this. This is a totally robust way and this is how in fact um Uh this is when people talk about like climate models or when talk about epidemiological models, this is what they're talking about. Um okay, so then Toby Ort comes and says, Huh you know what? My vibes are thirty percent I yeah, I see what you guys are doing over here. That's fifty years of grad student research, but my vibes are it's thirty percent. And so sure you can do your simulator thing, but my gut says thirty percent, and I'm not gonna tell the readers that I'm just making up a number from my head and then comparing it to somebody else's number that's come from a robust methodology that is well studied, well understood, is falsifiable has a lot of code in it based on like how people actually work in the real world, has been tuned do to tuned from other pandemics and other outbreaks so that we can make predictions and check if the model is wrong and if it's wrong then we tune it so that it's right. Um this is the methodological mistake I'm making. This is what I think is is is uh wrong. It's just it's calling it thirty percent and calling it thirty percent over here These 30% mean entirely different things because they came out of a different process. Um and so this is the thing I'm challenging, that this is just a mistake. Even if you don't sign off on any other arguments, surely you don't think this is a good practice Um I know people surely alarm, but this is a silly practice. You don't need to know much about math to know that discounting fifty years of research in a simulator Um should not be compared to just some philosopher sitting in their Oxford office thinking for a bit and being like, eh , thirty percent. Like this is these are different these are different methodologies. Um I'm just saying you can you can even say that the second methodology, the Tobyard methodology, is fine and we can argue about that. But even if you grant it's fine Speaker 2 Uh I have no idea how Toby Ord came up with his number at the end of the day. Okay great. I d great. I did not uh your characterization strikes me as Probably not the way he would characterize it. I don't actually know what he did, so I cannot no no no. Don't pull up the quote. I don't know what he did. I'm not gonna defend him. I can't defend him. I'm not him, I don't have the relevant knowledge Uh I think maybe you should go on a podcast with Toby Ord to discuss this uh because you don't like his methods. Oh well okay. I'm I'm sorry about that. But the th the thing is, um I I this j one really briefly Stephen. The No. Uh obviously your straw man, uh maybe you're real Toby Ord, I don't know. I can't say for sure. Uh but your characterization is ridiculously bad and should not be done, and also uh I think that you are drawing a fake comparison because partly because uh that is not how people generally come up with their numbers. They don't make numbers up and just say this is the number I'm going with. There is a methodology you can usually look at. Uh I don't think that your simulation is necessarily uh as great as you think, especially if you're trying to track individual things like how many people are gonna touch which doorknobs and which mail. At that point I think you're getting into the crazy realm of trying to track things that are literally untrackable because we don't have that site uh that sort of omniscience. But I just I find it oddly appropriate to that you bring up COVID, because as Ozzy said, the rationalists are gonna be running laps uh victory laps around COVID for the next decade because when everybody else was saying this is no big deal, you know, don't worry about it, when all the major institutions were saying you don't have to worry because our simulations say this or whatever, the rascists were the ones that were like No guys, actually this looks like it's a really big deal. Here's the reasons why everybody actually start preparing right now and they they were amazingly prescient on this. They've been amazingly prescient on a number of things, which is I think the reason why they get so much attention now. And bringing up COVID I think is a great example of how the ration the Bayesian rationalism actually will give you better results more often. Uh Speaker 1 so the author Vaslav Smiel wrote a book um in twenty two thousand six. um called Global Catastrophes Global Catastrophes and Trends Over the Next Fifty Years. He also predicted COVID. He predicted it earlier than the rationalists. He predicted actually the time band in which it would happen. He got that right. I think he said like 2020 with a um uh margin of probability or uh margin of error of about three years. This is what motivated Bill Gates to then start doing TED Talks about it. Um and he, uh Vasov Smiel, one, entirely rejects the rationalist methodology, as he says in his book. And two, he did it based on data. And you don't need to come up with credences because you can look at the historical record of how often you have large pandemic outbreaks over the last hundred years. So no, it is not the case that the rationalists were unique in their predictions. But They did get it right, and that's great. And how did they do it? I would claim they did it based on the arguments about lab leaks and all the stuff that Ord writes in I think his appendix F. Um I would claim that they did not get it by just coming up with credences. They came up with it by reasoning uh the way that I'm suggesting they do reason. They should reason. um which is non-probabilistically, uh uh coming up with arguments uh finding out that oh my god we do not have proper Uh pandemic awareness risks and all this. But uh your claim that the rationalists were unique is not true. Fast left smeal, anyone can fact check it. And two, to the extent that they did come up with these predictions, I would claim the probabilities didn't help them It just kind of uh distracted them from coming up with arguments, as Tobiar did in his book. Also, I want to quote from the precipice because I am not strong at manning him. Here's a quote. He says A common approach to estimating the chance of an unprecedented event with earth-shaking consequences is to take a skeptical stance, to start with an extremely small probability, and only raise it from there when a large amount of hard evidence is presented. But I disagree. Instead, I think the right method is to start with a probability that reflects our overall impression, then adjust this in light of the scientific evidence. In the case of artificial intelligence, everyone agrees that the arguments are very far from watertight. But the question is where does this leave us? Very roughly, my approach is to start with the overall view of the expert community that there is something like a one in two chance that AI Agents capable of outperforming humans every task of little. So he starts with one and two. Isn't that representative? No, we could talk about the AI expert community. Speaker 2 Which is a different option. If that's not true, then he was lying. That's not a problem with methodology. That's a problem okay, sure, mistaken, but that's a problem with the data. Speaker 1 Mistaken is completely different than lying. It's completely different. Um yeah. It's uh it's utterly different. It's not a tweak, it's utterly different. He's mistaken. He's mistaken in two ways. First way. Um just because He just starts by saying one and two. When you talk about AI experts, Speaker 2 to the best of his knowledge, according to you he was mistaken. Speaker 1 So I have been given a survey. I am, quote, an AI expert by his definition. I have a PhD in AI, I've published it in ERUPS, and I was given one of these surveys. Okay? Great. I'm telling you, as an AI expert, it doesn't make any sense to talk this way. Are you gonna update? No, because you're gonna do what everybody else does, which is cherry pick the experts that agree with you. Okay? I can give you fifteen different experts. I can give you Melanie Mitchell, I can give you Mike Jordan, I can give you Um myself, it's cringy as hell, but I beat the definition because people just talk about people with PhDs when they talk about AI experts. Um I can give you uh uh Ben um Ben Recht. um myriad of other researchers, uh Francis Bach, um, who don't think probistically, uh who don't come up with those stupid little credence and they would Just reject this whole uh methodology. Um when I was given the survey, so I was I took a survey, I am probably in one of these quotes, right? What was the survey? Um First, they said, okay, I want you to answer these questions. Here's a slider. You do not have a choice to opt out. You can't say NAN. You can't say uh I reject the whole method. You just get a slider between one and a hundred. Okay. Second, I was given it Speaker 2 answer Like if you just not return the survey. Speaker 1 And that that's how you get selection bias. Speaker 2 Okay, well then can you put it at zero? Speaker 1 No, not with the sliders I was given. No, you can't. Speaker 2 What was the lowest you could put it at? Speaker 1 Oh okay, well Speaker 2 if you a lot of people put it in the middle of the year. Speaker 1 So of course they wouldn't bound it by zero or one. For subjective things except stuff you can countermeasure. Second, I did this over lunch because I wanted a $75 Amazon gift card. I was like, fuck, whatever. I'll just slide, slide, slide, slide, slide, slide, slide, slide, slide. Um got the gift card, bought myself a hat or something. Um This is the evidence that he's using, I'm sorry. This is not at all comparable to counting asteroids. This is cherry picking a bunch of astero uh uh experts Speaker 2 So once again it seems like you have a problem with Toby Ord and you should go talk to him or publish a piece bringing down his book or something, because this this sounds like a specific Yeah. Speaker 1 No. I have a problem with the methodology. And second, I have published pieces. And third, I did get um William McCaskill to take down his pieces because they're making the same mistake. Um Speaker 2 congratulations. I am I'm glad you managed to yeah, to um make the discourse better. Yeah. Sorry, Stephen, you were saying something? Speaker 3 Well, I I think like to me, Toby Ord the the example here illustrates like the floor of where none of us will sign on as this was good thinking. Um I think that I think that what you're saying is that well no the Bayesian rationalists think that what Toby Ord was doing is close enough to being, you know, rigorous or whatever it is that we we're doing, so it's it's okay. Um But I don't I certainly don't like the idea of him kind of pulling a number out of out of his air out of the air and saying, This is the one, let's let's reason about this number then. Um Speaker 1 Which way he didn't know. Speaker 3 No yeah. the way that the number came to be on the page. Um so I wonder if like we want to find like a different example uh I guess I've got I've got two questions and and if they're both kind of side railing from what we're talking about then we can just ditch 'em. But like maybe we can think of a different example and I've got one ready of like here's what it looks like to reason with numbers. Um and then if we if we wanna May maybe that but also Speaker 2 yell at you that the numbers are bad because they were from a mistaken survey. Speaker 3 uh maybe what's wrong with this kind of approach from from his perspective, which I'm curious about. But the the other thing is like m if we if we do find the number chat getting kind of annoying, like How how how should we actually reason under uncertainty? Because like you know, so so let's say it's 2020, I'm reading all this stuff online from people that I think are pretty sane, uh that are saying, hey, this this COVID thing might be a big deal. Like, what do I do? Like, you know, do I pull you know, how much of my money and resources do I divert into what these people are saying might happen? Like So I I I guess I'm curious and and the other example if we want to just uh flag back to it later is like I was thinking Robin Hansen's calculation for why he started ever cryonics Um I don't have the numbers in front of me, but we can spitball them. Um for for someone who arrived at a probability w by doing math and not saying, well, here's my vibe. Um And so I I think that his his his Chronix math is much i his chronics number is much more sane than Ord 's uh uh P Doom despite his P Doom being in the wheelhouse of what people aver on average estimate. Um but yeah I g I I guess I'm also just curious like how how how a human is supposed to reason under general uncertainty Speaker 1 So if I may suggest a different example, but if you reject it, I'll go back to your examples. Um just a choice that we all have to make, like um what job to get, which school to go to, who to marry, like big life decisions, right? Um is that okay to use that example or do you want to focus on the COVID and the cryonics? Um I'm not trying to shift the the question. I'm trying to it's something that people have experience with and I think COVID is unique in a lot of cases. Like what what I do personally, I did nothing because I can't really do much. I can sit at home when I'm told to. Um I didn't have a lot of money as a college student. I couldn't really donate anything. I was not in a public policy perspective. Um so the COVID one is tricky because I think what one would do if they're in public policy is different than one would do if there's an effective altruist, different than what one would do As you're just a broke college student who's just trying to get by. So that's why I think that example is a bit tough. But um but yeah, so Uh so if if if we give a permission , Speaker 2 a lot of rationalists were not in policy or something, but did make it they did prepare in some way for COVID. Like I was just an accountant in Denver. Well, I mean I'm just saying like you did nothing and that's fine. That's you know, and that's what the vast majority of people did. But a lot of A lot of rationalists actually did do some things to prepare. And you could have, and that's great. Yeah, exactly. That's the big things we're talking about. Speaker 3 Like is it, you know, so some people went earlier than others, is kind of what I'm getting at. Speaker 1 Yeah, I mean um uh the so I didn't I didn't stockpile. Um I Uh so we want to focus on COVID? That's important to focus on COVID. Or can I use the other example as a elucidating step into it? Uh Anyash, I feel like you're You want to do that? Speaker 2 It doesn't have to be COVID. The the point is that you can have some beliefs about what the future is going to bring and of those expectations will guide what you do. Of course. Of Speaker 1 course. Yeah. But beliefs are not the same At no point did I say one should not have beliefs. Please. I said one shouldn't put numbers on one's beliefs and use numbers as a means of doing like a decision theoretic analysis with a little grid of like this, that, the other things. Speaker 2 Alright, so how should one come to beliefs? Speaker 1 Come to beliefs is different than reason under uncertainty. Do you want me to answer the reason under uncertainty or come to beliefs? Speaker 3 Can you explain how those are different? Um Speaker 1 one has beliefs for a myriad of different reasons. Belize is a psychological um phenomenon uh but that's different than how does one decide to act on one's beliefs, especially in a state when one does not have um definitive evidence um or conclusive evidence to guide one's decision into so one is a decision question, like how do you act? And the other is a psychological question, like how are beliefs formed and what should one do with one's beliefs? Uh so those are separate, right? Speaker 3 Okay, I see. I I was thinking because I like my my beliefs inform what I do. And I I think they do for everybody. Uh but yes, they're they are uh they're distinct from the actions. Speaker 1 Yeah, so um first let's just make sure that when I say the word expectation. And when I say the word uncertainty, I am talking about psychology. I'm not talking about probability. I'm not talking about um like expectations i. e. like probability times the utility m summed over all things that can happen. I'm talking about like uh I expect two um Continue having a great conversation with you guys even though it's getting heated and I'm loving it. Or I expect to uh uh to be uh annoying many people in your audience at the moment. That's that's just uh something I expect, and that is Something we all have. Plants have expectations. Um they expect that uh like when they put the little roots in the ground that there's gonna be water and they move the roots to find water. So there's inbuilt expectations. Speaker 2 Yes, let's go with the Speaker 1 same with uncertainty. Same with uncertainty. Uncertainty is not the same as probability, right? Uncertainty is there's a number of actions in front of me. I need to do something. Uh but right now I don't know what to do. I need to make a choice to to act, right? Um so I like to take not a decision theoretic lens on uncertainty and decisions, but I like to take an evolutionary lens on certainty and decisions. So getting a bit More to a question we haven't dived into too much, like what is critical rationalism all about? Critical rationalism. You could invert it, you could say rational criticism. So from the Pauperian perspective, the most important thing, the primitive So I'm saying some things You guys are thinking about it. And you're criticizing the living shit out of everything that I'm saying. And you're doing it rationally because you're like we're both trying to attract one another's arguments. Um we both are um uh geared towards truth. Um if I make a mistake in in uh something you've said, then I would correct it, you'll make a uh and vice versa. And Uh that's what the rational part means. Just genuinely trying to listen to your conversation partner and in particular listen to their criticisms of your views. So that's what critical rationalism is or rational criticism is. Now, when I have to make a big decision, so uh we can talk about COVID momentarily, I'll use as a stepping stone um deciding which school to go to. Um if the decision has low stakes uh i. e. I'm deciding between pasta primavera or uh Rubin sandwich at a restaurant. My criticism mode might be, had Ruben yesterday gonna have pasta? Because the stakes are low. The the outcome is is not a very important one, so I'm not going to criticize it for too long. If it's a very important decision Um one that like uh as Jeff Bezos says and I like to say is like a one-way door decision, i. e. once you make it, you can't unmake it again Right. Um then I need to really be getting as much criticism from as many different people as possible in order to start um Eliminating options that don't work. So it's not about putting a bunch of probability on the best option. It's about okay, I need to decide if I'm gonna go to UBC or University of British Columbia where I went, did in fact go, or um uh McGill, Montreal. So it's a decision I have to make. Um spent a lot of time thinking about it. I was like, okay, well uh If I go to Montreal, I'll need to learn to speak French. I don't really want to learn to speak French. Um and it'll be far away from my. So you think first in your head about like, okay, if I do this option, what's the problem there? Oh, that's a problem. That option is eliminated. And then perhaps you're left with two or three. And then you start talking to close confidants, friends, people who um understand the stakes, understand the outcomes, and are helping you reason through it by the process of elimination. And typically one can criticize a set of options for long enough such that eventually there's just one option that has withstood the most amount of criticism. And it may not be perfect. It may not be the one that's gonna definitely optimally give you the right outcome, but it's the best you got. It's the best you got. Um and that is how Human beings actually do, in fact, make decisions. They talk to people, they brainstorm, they do a little scrum, they do a stand-up, um, and they just try to be like, okay, here's what I'm thinking. What do you guys think? Is this wrong? Is this right? Oh, it is wrong. Okay Great, I didn't even realize that. Good point. Okay, let's try this other thing. Um until you're whittled down with one option standing. Um on the COVID example Same process. Uh for Bayesians, a lot of them are just doing the critical rationalism thing that people do already. But then they add probabilistic stuff on top, uh which just gets in the way and it makes the criticism a bit harder. But in fact they're just reasoning, they're talking, they're blogging uh an unlimited amount on the forums, they're responding to the comments, and um if you do this process long enough Uh then you can arrive at some of the really good parts of Toby Ward's book where he points out that there's like a lot of uh, for example, sh bad safety practices involving like uh lab ceiling and and lab leaks and and that kind of deal. But that's because he wrote his book, he talked to a bunch of people, he then presumably sent it to a bunch of folks to get edits and get critiques, and he sent it to some microbiologists and some blah blah blahs. Um this is critical Rationalism and it's evolutionary, I said it's evolutionary. It's evolutionary in the sense that uh you can roughly map the idea generation stage to the random mutation stage of evolution. And then you can map the criticism stage to the environment selectively retaining those ideas, those genetics or those genes that withstand or most robust to criticism in the case of decisions or just the environment in the case of evolution. So you can think about this process evolutionarily. via that that mapping. So that's how humans in fact do make decisions. And that's how I would in fact recommend people try to make decisions. The more important the stakes, the more you should get criticism from people you trust. Speaker 3 I liked that that that walk because it it lines up a lot with like my model of how we have scientific progress. Speaker 1 Exactly. Um Speaker 3 what do you ma what do you imagine patients were doing differently in like the which school do I go to calculus? Like Speaker 1 they would make little decision tables on top And then sign utility values. They do decision theory. So they design the probability of and it's just irrelevant. Speaker 3 I m maybe like the the perfect rationalist in a sphere would sit there with uh in a you know spherical vacuum would would would sit there with pen and paper and and just do math and come at their answer that way, but like Everyone I know, they they do both, right? They will they obviously talk to people to inform their decisions and do homework. But at the end of the day they w pe you know, whether they're they're crunching numbers or not, they are doing an expected value calculus where it's like How much how much will I hate learning French? Man, that will be a huge drag. That's that that's gonna be a big point against me going to that school. Um Speaker 2 all my friends are going to this school. Yeah. Put different weights on different things. Speaker 1 You don't have to? Speaker 2 Yes, but everybody does. No, no, no. You said you put different weights on different things. You did not consider the difficulty of learning French to be the uh the exact same Same weight as whatever other consideration like what they certainly have to do. Speaker 1 Prune so Speaker 3 sor let me just make sure I I I actually want to understand. I'm not I'm not arguing with you yet. Um like the Is yeah, running wondering with the school example. Like if if there was one that's like you know, learn French but uh r uh you know, so school what was it, the Montreal school Speaker 1 Yeah, McGill or UBC. Actual decisions actually have to make. Speaker 3 Right. And so now I'm I with the actual decision you actually have to make, I'm gonna make up a bunch of stuff with it. So , but all of your friends are going to um uh McGill Uh but you don't want to learn French. Like, is is that a deal breaker for you, or does the fact that all your friends are going there suddenly put it back on the table Speaker 1 Yeah, I mean there's no uh formula that I can give you. Um that's why there's no weight that I can assign. It would just be like Um you know well John 's going and Sheila's going and Michael's going and they're so sweet. So yeah, you know what, that's more important to me than the French thing. I'm just gonna forget about the French thing. Speaker 2 Oh my god. Speaker 1 No, the the weight is zero or is one. If I say forget, that's prune, right? It's not a wait. A wait Speaker 2 But you wouldn't have forgot about it if it was just Steve. If it was uh Speaker 1 waiting and pruning? Do you understand the difference between waiting and pruning? Speaker 3 I didn't actually until just now. I thought that when you're waiting for the thing. Speaker 1 And additionally, what are you multiplying? You're trying to multiply the made up continuity. Speaker 2 No no no, you're the one who introduced numbers into this. I'm just talking about waiting, as in in no, we were talking psychology from the very beginning. We said let's just talk about psychology in this things. We aren't talking numbers, we aren't talking probabilities. Speaker 1 Uh when I talked about when I talked about evolutionary decision making, I did not include numbers. Um you included numbers. Exactly. When you talk about weight , Speaker 2 a weight is a weight. Sir Sure. We can use the mathematical definition of weight if you want, but we weren't talking about mathematics. You very specifically said we're not talking math now. Expectations isn't a psychological thing And uncertainties in the psychological thing, we're not talking about the maths. Yeah. And so in normal parlance, people will say, you know The money is more important to me than the French thing, so I weigh the money more. And they don't mean like I am putting a number of seventy-seven on the money and a number of thirteen on the French. They mean it waves more heavily in my psychology. So don't be pulling this bullshit where you're trying to tell me I'm putting numbers on things when I am use new when I am using it Colloquially, the same way we were talking about it until just now. So colloquially, the French thing was a really big deal for you, and some friends are, you know, a small deal, but like all the friends are actually a big deal. So, yes, everybody puts different weights on things. The fact that like they don't serve your favorite lunch, maybe that's a teeny little thing to consider, but it's not really that big a deal. Uh, and that that is Speaker 1 Okay if you b so okay, so I I didn't understand what you meant by weights. I thought you were talking about like decision theory, i. e. weight equals I mean Speaker 2 that's because you've been modeling me as a straw Vulcan this entire time, and I am not, and I don't think Speaker 1 there are mischaracterizations if I if I did so. Um Speaker 3 I th I think that you're a mathematician and When you use mathematical language it's it's typically in a precise uh and known academic way. Uh we We here we're we're we we're not mathematicians. We're we're we're just we're throwing words around. No, yeah. So I think I think that's all that happened. Maybe maybe using the word weight there is more of an Americanism, not a Canadianism. Speaker 1 No no no I I use it I use it too. Base. Gotcha. And decision. Okay, cool. And a weight means a probability. Um yeah, so uh I have two options. Um McGill or UBC. I consider a whole bunch of um criticisms of both options. I think really hard, I'm not putting any numbers on it, and eventually I rule out an option. I don't add them, I don't average them, I don't I so to use the word weight colloquially just means I have determined that this criticism is more valuable than this other criticism and it's decisive and it has now eliminated an option So it's a process of elimination. Um that's all fine. That's great. Uh so I I guess um are we in agreement then or 'cause I got the impression you're criticizing me. Speaker 2 No that's a perfectly fine way to do things. Like I I don't have any problem with that. You seem to have a Speaker 1 non-Bayesian that's a non-Bayesian way to do it. Uh Speaker 2 I think that it is at least approaching Bayesianism in that. Okay. Very well. But uh I I have no problem with that, you know, with that being done that way. Like most people do not go through and do uh equations when they make any decision or make any belief. Like the entire process is very informal Uh and you seem to be upset that some people might make a table where they put like a water. Why why do you why do you Okay, I I guess I shouldn't have t used the word upset. Um Why do you do this hand gesture and make a scrunched up, you know, this is so yucky face when you say that some people make a table when they make these decisions uh and add a number to it? Speaker 1 Because in the best case it's a superfluous and unnecessary step. Or no, I can go stronger than that. In the best case, it's like a Uh internal reasoning tool like I've made like a pros and cons list before, like a tally, like what for sure it could it could be like a reasoning tool. Uh in the worst case, it leads to Bentham's Bulldog. In the worst case it is um okay there are this many shrimp and mosquitoes And I have this small of a probability that they're conscious, but multiplied by my probability of human beings being conscious by the smaller number, then in expectation we need to cancel all malarial programs and take the side of the mosquito. That's in the worst case. Um and that is extremely dangerous, literally gets people killed. Um and additionally it's totally unnecessary. Um we're talking about this whole thing because you guys had said Vayton, if you're trying to get rid of numbers, how on earth can you deal with uncertainty? Like how can you make decisions in an uncertain environment given that you've just told me you can't use numbers? And then I give you an answer and then you say, Well actually we weren't even using numbers in the first place. to get a lot of things. So that Speaker 2 I would like to protest to say that we did not in fact say say those things at all. Yeah y you keep bringing up Bentham's Bulldog and I I want to again point out that people argue against Bentham's Bulldog and consider him a crazy person and Any any place will have fanatics. Like I you personally I think have said you dislike the crit rats because they have gone crazy on Twitter or something Yeah, yeah, okay, so uh any any any idea will have fanatics that and you cannot judge an idea by the craziest people that subscribe to it. But No Speaker 1 no you can judge an idea based on the amount of crazy people it produces. In the case of crit rats, my criticisms are about the ideas so associated with libertarianism. And so I have all sorts of criticisms about libertarianism, precisely because it leads to a lot of insane ideas. Um, so too with Bayesianism and the effective altruist and the rationalist world, because this particular set of ideas leads to crazy And so, yes, you should criticize the idea if it reliably leads to crazy. Hmm. Speaker 3 And so you think that maybe Speaker 2 I don't disagree with that at all. Speaker 1 Nice. Speaker 3 Yeah. So is then the concern that like the Like, the Bayesian crowd is ill-equipped to combat exactly where Bentham or um I should say Bentham's bulldog, Bentham was fine. Well actually Beth Beth them Betham was talked about a lot of expected utility as well, actually, so that's why Bethesda will just pulled a whole for him. But like so yeah, but it doesn't give us firm grounds to argue with. And I guess that is something that is worth exploring because like um like I I I'm not uh Speaker 1 Roko 's Vasilisk. Speaker 2 People have panic attacks. Speaker 1 People have panic attacks about that. Speaker 2 I I'm fairly anti-EA myself because I do think it leads people to bad places. Um it also leads to good places. Speaker 3 What I'm trying to say is like I've I've uh I I I I don't necessarily I don't really classify as an anything, including an EA, but I'm sympathetic with a lot of the arguments. And like I'm I'm sympathetic with the core reasoning of like You know, we can't be sure that animals don't suffer. In fact, on you know, it it seems, without using numbers exactly, more likely than not that they probably do suffer something. Um, you know, they they act the way that we do when you hit them, for example. Uh something like that, right? Well right, the the ones that do. Um but like uh Where I think there's missteps is like where people do try to start assigning numbers. And it's like well a a a cow is one sixth as intelligent as a person, therefore they matter one sixth as much. That to me always f seems like it falls apart Because that number has to come out of basically uh the ether. Um but the idea of like we ke l let's be humble in our um our assumption that like humans are all that matter. Maybe that's not true. Maybe there are other kinds of beings that that have concerns worth worrying about. And so like The way that I deal with that is I eat less meat than I used to. Um I don't eat no meat and I probably should eat less. But like so I I guess I wonder um If we're I feel like there's a lot to preserve there. Speaker 1 Totally. I agree. Speaker 3 So how how do we keep any of that without or with with uh with this with the crit rat Speaker 1 framework? Um Yeah, well so uh actually let me go stronger than what you said. Uh I think that some crit rats go way too far with a misunderstanding of the framework and conclude that all animals are not conscious and only human beings matter and I think they're crazy. Um and so uh Uh when I think about animal consciousness, like all we really have to go on are basically intuitions and then some light experiments. So let me be specific here. So roughly my model of like animal consciousness is the following Um mammals, probably conscious. Um go down to like squirrels and ro squirrels or like big mammals, like macroscopic uh charismatic megafauna, dogs, wolves, cats, probably conscious. Squirrels probably conscious. When you get down to like ants and like fish I don't really know. And then when you get down to like plants and stuff, I'm like just gonna assume that they're not conscious. And then you can run certain experiments. So for instance, one thing you can do is see if like Like um which animals respond to painkillers? Um and do they do like uh dogs do a thing called uh pain guarding, i. e. if they have a wounded paw, they'll limp on it. Um they'll limp on it, uh because they need to guard it from from um external uh uh stimuli because of the the presumably the sensation of of of unpleasantness, call it suffering. Uh and then if you give them like morphine and stuff that works on human beings, they'll do it a bit less Um insects you can pull off all their legs and they'll just keep running around in circles and they have no w idea whatsoever. Um so in the case of Uh or if yeah. Yeah, no, no, totally fair. Um so basically we know nothing about consciousness. We know zero. Um but we have evolutionary reasons to um tentatively operate with the assumption that other things that have been evolved have likely evolved similar um forms of uh like environmental protection that could likely map on to like sensations of of pain or or um uh joy in the case of say a dog like wagging its tail. Um and then the closer you get to like us to evolutionary tree, like chimpanzees and stuff, like I think it's very likely, I'll say likely without shame or embarrassment. I think it's And then you can just make decisions based on that. So like we just we don't know much more than that. But I'm not gonna like abuse my dog. Um I am a meat eater. I the it's probably the the uh thing that is just hypocritical of me. Um but I frankly don't care enough to uh to to not eat meat. Um but I I can't really argue against the veg yeah I can't really argue against the vegetarian. I think the vegetarian has a morally superior position than I do. Um and I I'm a I'm a hedonist. I 'd love to hear your arguments against it. But uh but but this is this is about as good as you can do. And then as soon as you start trying to do like But we're this uncertain, but there's so many of them. Oh my god, there's so so many of them. That's when you start doing Pascal's wager kind of thinking, even if you don't assign probabilities to it, and that's where the mistake comes because you can't multiply ignorance by a lot of it and get knowledge coming out. You just you're just ignorant. You doesn't matter how many mosquitoes there are, you're just ignorant about it. So um don't try to Trick yourself into being precise about your ignorance when you just don't know and so don't pretend they know and act based on that. Speaker 3 I think that people trick themselves into the false false sense of certainty a lot, like Bentham saying a human is worth seven B's or whatever. Like Speaker 1 this is a mistake because of the probability calculus when Bayesian isn't like you can't do Pascal's wager style stuff if you don't do a uh Bentham Bulldog style analysis. Speaker 3 But I guess I'm wondering can you can you still kind of reason yourself into the right posi into the right actions if uh like you're not I guess you can be you can be motivated by And I'm not using this derisively, like your vibes. Um instead of being motivated by made-up numbers. Speaker 1 I think we all are motivated by vibes. This is what um uh was it David Hume who pointed out that We um uh reason is forever the slave of the passions. Uh it is vibes. Um we start with vibes, we start with common sense, we start with intuition, and then we can sometimes turn that into something a bit more objective. But um Uh I'm not against vibes. You have to start with vibes. You have to start with intuition, um of course. Uh but let's just not pretend it's anything more than that. Let's just be humble that sometimes that's all we got. And so okay, we have to make an action and like If uh I think if we could come up with a world that replaced factory farming with like lab-grown meat, that would be a much better world. Totally. I would support um organizations that are working on this problem. Um Uh I think it's great. I think it's a I think factory farming has a lot of problems with it. You don't need me to go through them. Your audience will be familiar. I'm not against any of that. And you don't need to and most vegetarians I know Aren't Bayesians because you can just get to that position without having to put numbers on stuff. Speaker 3 So I think I think this is great because we found some spots where I think we're in total agreement. Where it's like in some of these kinds of questions it's not worth assigning probabilities or it's counterhelpful because you fall into all these these uh traps that you set for yourself if you try and put numbers on them. But I think there are cases Where maybe maybe the numbers don't help. But well so I think that that might be worth talking about and this might get us back into like AI safety. Where it's like 'cause you know, if I if I'm if I'm wrong about Speaker 2 in every case numbers help a little bit. Like I I do think that um Everything is vibes. Uh, and that is, I mean, that's where we start with, and more importantly, that's what motivates humans, right? Humans aren't motivated by numbers, they're motivated by their vibes. Um but I think having some numbers gives you a better idea of what is actually happening, uh as opposed to being like, yeah, I I kinda I kinda like animals and I think we shouldn't hurt them. Uh Do you know how many animals are being harmed? Do you know how bad the harm is? And does that matter to you at all? Like if it doesn't, that's fine. Like I eat meat, I think it's actually very important, uh, to eat meat and that a lot of the discourse is kind of just awful to poor people and um resigning poor people to have bad health. Uh and I didn't know that's a good idea. But anyways, uh yeah. Um Speaker 1 long lab grown Beef. Speaker 2 Uh the Yeah. Nonetheless, uh putting some numbers on things makes it uh easier to reason about things. And I think also importantly Yeah, yeah, sometimes, right. And importantly you sometimes want to know if you should change your vibes. Like I've got Speaker 1 criticism is for Speaker 2 Sure. And that's also what uh analyzing things with numbers uh can do. Like my Speaker 1 problem lends to that, not all problems lend to that. Speaker 2 Sure, yeah. But if my health means that like a cow is inconvenienced, I'm okay with that. If my health means that seven babies need to be tortured and then sacrificed, I'm not okay with that, right? Like we this is we actually have fiction about this. Uh so knowing the difference is I think important and I would want to change how I feel about things based on what I actually know. And The tools are useful for that. Yeah, yeah. Please go ahead. Speaker 1 Yeah, so let me um let me uh highlight where I think using numbers is valuable. Um uh cause there are a lot of places where numbers are valuable, of course. Um you describe me as a mathematician, that's kind. I don't think of myself as a mathematician. Um but I d I like math. I've did a lot of math in my PhD. These days I'm just more programming, but um I like math. Math is great. You want to use math for all sorts of stuff. Um typically you want to use numbers when you are measuring something or counting something. Measuring gives you a continuous, what's the velocity, what's the weight, what's the speed, or counting how many people go to the hospital, how many um people will be affected by COVID lockdown, for instance. Uh let's say you go to a casino, let's say you are playing poker. Um there I do actually think in terms of the expected value calculus. Um it's not perfect, it's imprecise, but why is that circumstance Valid. That circumstance is valid because it has been set up by the casino to operate under the laws of probability. So we know what the distribution is for the cards, we know um what the distribution is for the um uh any particular hand given the fifty-two cards, etc. Um and I may not have perfect numbers, but uh I'll t I'll totally think like, okay, I gotta Um I've got a full house and yeah, this is a risky bet, but uh if I do it, uh I could win a lot. So that that's expected value. style reasoning, right? So you're coming up with some vague probability, you don't really know exactly what the true probability is. Um and you're coming up with some utility. Here the utility is quite easy because you can use the pot in the center of the table. Um and then you're doing some sort of rough kind of like approximately vibey um uh uh calculation in your head um that one can get more experienced with over time like uh live bowery for example great poker player um And that's great, that's totally fine. Because you're using expected values and expected value style of reasoning in a limited domain where we know that the laws of probability actually operate. This is not the case with the long term future. Um when we come and start talking about factory farming or let's talk about COVID. Um Yeah, I mean if you are a policymaker, you absolutely want to count the number of people who are going to be influenced by your policy, of course. Um if you are trying to figure out if you should focus a lot on like um uh beef or pork say uh in terms of what you're gonna grow in a lab. Yeah, of course you're gonna want to count how many um cows or how many pigs. Um of course, but if you don't have things to count Don't use numbers. If you don't have things to measure, don't use numbers. Use arguments. Use reason. Use criticism. Use brainstorming. Use non-quantitative things. to think about things that are fundamentally not quantitative. But if you Speaker 2 use numbers to compare things? Like if I have uh two light bulbs and I don't know how bright they are, but I can tell someone that light bulb's like a three out of ten and that light bulb's a seven out of ten They'll get the impression that one is much brighter than the other, even though I'm just kinda making up numbers to give them an idea of how I perceive the light bulbs, right? Um Uh Speaker 1 you can also just say this one's much brighter than that one. What is the other one? Speaker 2 That doesn't help nearly as much as like saying this one's like a four versus a five, whereas this one's a one versus an eight. Like that. I think that is when people do the informal Bayesianism thing, that is a lot of what they're doing, when they're putting like their numbers on how important something is. Speaker 1 As soon as you create the numbers and you put them on a piece of paper, now you can start doing integrals. Now you can start putting it through uh two hundred years of Probability mathematics. And by the time you've done all your calculations, you've forgotten that it's just a vibe that you put onto the in the beginning of it. But you've put a vibe to Speaker 2 you have, but I certainly haven't. Speaker 1 Sam Harris did. All readers of the precipice who didn't know the stuff I was just saying did. This is the fact that the precipice information is new to you means you forgot that, or at least you didn't know that. Uh he just uncritically swallowed the comparison between a one in six chance of e like X risk killing us in the next decade compared to um a one in one trillion chance of asteroids. The mistake was not realizing that in the first case it's a subjective probability. I it's just pure vibes. In the second case it was objective. I use counting stuff. Um looking for the first time. Speaker 2 I want to again I we don't need to get into this again, but I'm just gonna push back on the pure vibes. Uh that was actually based on something. It's not just somebody decided I think one and six is a good number. Speaker 1 It was not measuring the geological record, it was doing stuff. Speaker 2 Yes, obviously not, because there was no AI in the geological record. Speaker 1 Well I'm an AI expert and I'm telling you that it doesn't make sense. Are you gonna update? No, of course not, because everyone's gonna cherry pick their experts. Right. That's it's it's it's it it's if your oncologist said, you know, I just got a lot of vibes from all my fellow oncology experts, and they kinda said you have a one in three chance I would not take that very seriously compared to looking at some of the things that we're going to do. Speaker 2 Which is why that's not what happened. But Speaker 1 it is exactly what happened. Yeah. You know what? Yeah, we can move forward. Speaker 2 We don't have much time left, but you mentioned a few things, uh long termism. And I wanted to hit that for a little bit, like what is this long-termism thing and what is the problem with it? Speaker 3 And I'm I'm really curious, you said you got uh Will McCaskill to take down an essay. Speaker 1 Yeah. So he did not he did not credit me. Um but uh so five years ago I wrote a piece called The Case Against Strong Long Termism. You can Google it uh it's on my blog, I posted it into the EA forum, it got a lot of buzz. Speaker 3 What's the name of your blog? Speaker 1 Oh, it's just my personal academic website. The last post was like two years ago. It's not a regular blog. It's uh I'm annoyed and I need to write something, and so here's a place to put it blog. But um we can share it. Yeah, we 're Vmasrani. github. io. Um we can put it in the show notes. Um Speaker 3 cool. Speaker 1 So I was critiquing his uh uh paper. The paper was titled Um The Case for Strong Long-Termism. Um and then by the time I got so I wrote uh four blog posts because I posted it and then the EA forum lost their mind and then they gave me a whole bunch of criticism and then I responded to it in a series of three other blog posts. By the time I got to the fourth blog post, McCaskill had taken down his draft. put up a new draft and removed every single quote that I had done in the first draft. Um and then I got verification that they knew about me and were uh worried by my arguments, shall we say, from um I don't know if I could say his name, um someone who's high up in the EA world who uh I know who was a research assistant for uh Toby Ort, so but I I won't say his name only but he wouldn't potentially be uh known too to some of your listeners. Um I agree. Speaker 3 and they're aware of it, it seems like throwing in a throwing a a bone to s e even just pointing out like, you know The first draft of this uh was heavily reimproved thanks to Vaden 's uh contributions of of or critiques of my position. Speaker 1 He was not happy with the stuff I said. But um , Speaker 3 lame. Speaker 1 Yeah I'm assuming you're familiars. Right? Exactly. A little bit of rational criticism, right? So I don't know if you want me to explain long-termism. I'm assuming your listeners know about it, know what it is. Also, maybe just like Speaker 2 There may be more than one thing people think of. So uh what would you what what were you pushing back against here? Speaker 1 So the uh he was arguing for a thing called strong longtermism and the main thesis of the paper that I was arguing against, so this quote it, so uh this is McCaskill and Hillary Greaves. Um and so they say by the option whose effects on the very long run future are best. What we mean is the option whose effects from the future from time T onwards are best, where T is a surprisingly long time from now, say 100 or 1000 years. And this is a key thing. The idea then is that for the purpose of evaluating actions, we can in the first instance awfully simply ignore all of the effects contained in the first 100 or even 1000 years, focusing primarily on farther future effects, i. e. It doesn't matter if you hit your wife, it doesn't matter if you abuse your kids, because those are near-term effects, those are short-term. And it's not gonna positively or negatively affect the long, long, long-term future. You can do whatever you like, uh as long as the consequences of your actions don't have any ramification. to people living a billion trillion years from now. I claim this is morally psychopathic, um, but it's dressed up in Bayesian language. Um and so I can quote that if you like, but But that's the main thesis I was arguing against. The idea that all of your actions should only be evaluated based on how they influence the 100 billion year future from humanity and not influenced based on how they affect the present right now. That's the core thesis that they are pushing forth, and that I find to be morally repugnant to use a terminology that the uh EA people like Speaker 2 so I will completely agree with you that that is absolutely morally repugnant and I'm totally on your side, but also I'm gonna Say that I don't believe that's a that they would agree with your representation. I don't know. I I haven't asked them if they agree with this. I just heard you say this for the first time now But I would bet that if we were to ask them, they would say no, that is not at all what we meant by that statement. Speaker 1 That is the thesis that they were defending. That Speaker 2 long term isn't it. Speaker 3 You were you were reading a quote, right? Speaker 1 Yeah, this is a quote. all of the effects contained in the first one hundred or even one thousand years, focusing primarily on the further future effects. Short run effects act little more than tiebreakers. This is the thesis that they're pushing forth. Speaker 2 And I think if you went up to Will McAskel and you s and you asked him, so this means it's okay to beat your wife as long as it doesn't make the far future worse off, he would say no. That's just a defended random guess in the dark. Speaker 1 I mean he wrote the opposite. That's what strongism means. Speaker 2 Yeah, uh sure. Maybe that is the case. I didn't read the fifteen page paper. I would bet that it was some formality that they're talking about something that uh does not apply to wife beating and as some Speaker 1 No, but it applies to exclusively like the whole point of this whole device is to funnel everyone's charitable donations away from stupid causes like Reducing malnutrition and blindness in developing countries or eliminating malaria and other communicable diseases in Africa and Asia or school-based deworming programs Everything GiveWell was doing is basically what they were attacking, and they're like, no man, take all of that money and donate it to us so we can write scary blog posts about how AGI is gonna kill everybody and we can buy a bunch of Stanford grads laptops. That's the point of this thesis. It was to reallocate funds. by using Bayesian arguments, which I can go into, but I know we're running out of time. Speaker 2 I also disagree with that. I also think that what Givell is doing is a lot of good stuff and you shouldn't defund all of it to in order to work just on AI safety. Um again I wouldn't I think you sure uh I d I don't know. Maybe I am. I don't I I would have to read more about this. I don't think he would phrase it as writing scary blog posts. Probably he means something else. But uh regardless, I would agree with you that you shouldn't defund everything GiveWell does in order just to focus on AI safety. And uh I think you could have just said that instead of saying it's okay to beat your wife, man. You could have said this in a slightly less uh Speaker 1 No, I want to make the point clear. This is the same thing. Speaker 2 What is even the point about talking about him if he's just gonna say that if he's just gonna basically lie about the opposition? I understand that. Speaker 3 Um and there I wanted to that sh that that Speaker 2 No, no, I'm into puns now. I'm turning over. Puns are my friends now. Speaker 3 Great. Um Yeah, so like the the text of his of his essay didn't say it's okay to punch your wife, but like the subtext did, where it's like look, the only thing that matters is securing the future. And to the to the extent that this is why I'm like a I'm a bad ist at anything. Like I'm I'm I'm not an effective altruist because I I agree with the small like I I I'm not I I don't agree with any of the strong claims really any of the positions that I that I like. Um but like take uh You know, like you know, she don't like Peter Singer for reasons that you've used very similar language to Baden using here with uh Willem Caskell, where it's like, Oh, Singer's saying Like, every dollar that you spend on something that isn't a uh a life necessity is, you know, l uh money you're or money and food and life you're taking away from people who really need it. Speaker 2 Yeah, yeah. Me and Singer could punch each other, then one of us would get punched. Speaker 3 And so probably the eighty year old man would would go down in that fight. Speaker 2 Is he eighty? Speaker 3 Yeah at least. Speaker 2 Well if me and Singer at the age appropriate level could punch each other. There we go. Speaker 1 You should put him in a suit and drown him. Speaker 2 Right? I wouldn't take off my shoes for him. Speaker 1 Yeah, exactly. Speaker 2 And then when you phrase it that way, I'm like, oh my god, what's even the point? It's like he's he's not arguing for wife beating. Speaker 1 This is a conclusion of the thesis he's arguing for. And yeah, I of course I want to draw a attention to a conclusion that he wouldn't put in the paper, but this is why it's such a horrible thesis, because it leads to these kinds of conclusions. Speaker 2 One of the things I like about the Bayesians is that they do have a strong norm of taking the other side charitably as much as you can. Like eventually They just have shown to be in bad faith enough that you don't take them seriously anymore, you don't engage charitably anymore. But like, at first you try to engage people charitably and you you don't Speaker 1 Presumably the Bayesians appreciate reductio ad absurdum Right? Yeah. Speaker 3 I I'm I'm a big fan of it. And like you know, like Inyash, you know, Peter Singer isn't arguing against buying your children birthday presents. It's just a logical consequences of his arguments. Although Singer is unique in that he's been doing this for you know, sixty years and probably actually did write an essay about why you shouldn't give your kids birthday presents. But um Singer is is uh special and I think that he uh really understands all the corners and nooks and crannies of his arguments rather than just makes the thrust. But like uh You know, l we we we could spin it to um Speaker 2 I mean the thing is I've seen people that I care about derange their lives by taking singer so seriously that it hurts their lives and that's why I'm fighting back against Singer. Whereas the only people that I've seen that are like deranged by um Bayesianism is Bentham's Bulldog. And even so he mostly seems Speaker 1 Talk to anyone who's had a panic attack due to Rokos basilisk. That 's a good idea. EA forum post after EA forum post to people who are just torn apart by the long termist argument because they don't know how to get around it. They don't know how to get against this Pascellian wager style reasoning because they're so um inculcated into the singular way to think about everything, i. e. via Yukowski 's laws of rationality, i.e. credences on everything, i.e. multiply small number by big number and reach apocalyptic Dangerous conclusion because they don't have the the tools to think differently. Um and uh so I can give you a number of people who have have uh been negatively impacted. By Bayesianism in the exact same way that you have realized that people are negatively impacted by a singer um and are similarly interested in pushing back against it. Speaker 2 I I really kinda want to have you back on at some point to talk about the long-termism thing because I think um I I mean I know that I agree with you on this, but uh I am still a Bayesian and very strongly so And like I think there are very good reasons to dismiss the strong long termism that, you know, nothing matters except for the far future from within a Bayesian framework. And so, you know, I think that would could be a fun conversation. Speaker 1 Love it. I'm happy to come back anytime. I can't wait. It'll be fantastic. Speaker 3 I think that'd be a lot of fun and and valuable. I find this really informative. Um I don't wanna I'm not ready to sign off just yet. Like the thing. Speaker 2 I do think long-termism is important to a degree, but not like to the extremes strong degree that you're talking about. Speaker 1 I think thinking about the long-term future is definitely important. That's what like, for example, the US founding fathers did. And that's great. And you can do that kind of thing, and that's amazing. But you don't need to do it this way. Speaker 2 Right. Speaker 3 Yeah. Like I've talked to people at Lighthaven that, you know, they they heard about AI Doom and they're like, yep, I quit my my job and and left school and moved here to work on AI safety. Oh, you're not doing that? Man, you've got weird priorities. Well actually some somebody there said that to a friend of mine when she had said, I can't just quit my job and move here to do that because I have a career and I take care of my daughter And they're like, Well that's you've got weird priorities then. So it's like some twenty year old, uh, to to somebody with like responsibilities because they're they're I think the you saw about uh long termism for AI Doom and they've they've they've swallowed the same pill, just different exact different coding. Speaker 1 And what's the similarity between all of them? It's always the same argument. Probability multiplied by bad outcome gives you decision relevant information. Speaker 2 But I find this interesting because like this was specifically warned against in the sequences with Pascal's mugging, where he said like you cannot let the tiny probability multiplied by a huge number affect you. Um and like there is a Again, within Bayesian framework, how to not be mugged by this. Speaker 3 So I think there 's a Speaker 1 idea of my crowd. Speaker 3 Instead of saying, well, on the infinitesimal chance that AI is created and goes rogue, we need to now orient our entire future around preventing that because the the losses from it are cataclysmic, that's that's the mugging where it's like You know, we we take the one and one trillion chance times all of the f all of the loss of the future. But if we think uh Well I I shouldn't have used one in a trillion chance, or I should just say infinitesimal. But if we take the very reasonable chance that this go could go badly in the not too distant future Um that's no longer a mugging. It it's just it it's basically being warned of high crime. I'm I'm ruining the analogy. So it's Yeah, I th I think that th they're they're importantly different. Um but yeah, I think that it would we we we we started three recording sessions for this with this episode, just some uh inside baseball. So it's I don't have an intuition of how long we've been going actually, but now looking at the clock it's almost three hours. Speaker 1 Uh two hours two hours. Speaker 3 Yeah, I did so much. Speaker 1 Understood, understood. Cool. But this was a great fun. Yeah, I'm happy to come back anytime. This is a blast. Very curious what your audience uh has to say about this. If I can plug, check out Increments Podcast. Uh we're on YouTube and all your various podcast uh providers. And um I think the Uh episodes that your listeners would be most interested in. Uh we were on Doom Debates, but on our YouTube channel, if you go to the playlists, there's a whole series on the philosophy of probability. Where we start right from the scratch um and we go all the way up to like Doom debates and stuff and so we start from foundations um and then slowly uh build up each concept. And so that would be a good place if you want to dive more into how probability uh works and the various ways that different communities use it. Speaker 3 I'm really interested to do that. I found this fascinating. Like I I think that uh Well, I mean, I guess I don't fully I I don't fully understand all your position uh your all your position on all of this just yet 'cause I haven't uh I haven't listened to the to the um to the podcast and read all this cool stuff. But I'm inclined to. I found this interesting and I feel like we disagree uh less than you think we do, which means that I think I'm missing something because uh you know, you're the one advocating for the other position, so Uh but from where I'm at, this th a lot of this makes sense. Like sometimes it's inappropriate to to plug in numbers. And uh I'm just rehashing. I guess the point is is like um I found this valuable and interesting and I plan to think about it more and read all about it. So Speaker 1 Amazing. Speaker 3 Uh this is great. Speaker 1 This is fantastic. I had so much fun. Yeah. Speaker 3 Thanks, man. Speaker 2 All right. Thank you. And uh we'll talk to you again in the future, I'm sure. Speaker 1 That sounds good. Hopefully the short of the future. Speaker 2 Yes not in over a thousand years. All right, bye guys. See Speaker 3 ya. Speaker 2 Okay, that was I was not expecting that. Speaker 3 I wasn't sure what to expect and so I got basically what exactly what I was expect expecting. Speaker 2 Okay. I uh yeah, I was I Yeah, did not expect to come in quite so hot, but uh once we got going I was like, okay, well sure, let's do it now. Speaker 3 I actually thought it would be kookier Speaker 2 Mm-hmm. Here we go. It was it was fine. Yeah, it was great. Yeah. Uh this is our first YouTube episode and it's gonna be cross posted, so I think we should say for the people who are listening to it from Fade and Speed Uh you can listen to a lot more of our stuff. We have over 250 episodes of things. Not all of them like this, but some of them like this. He mentioned the Bentham 's Bulldog episode, and that was uh yeah, that was a whole thing. Uh that was less Speaker 3 fun. Nobody learned in that one Speaker 2 Yeah, yeah, that's true. Speaker 3 But yeah, we've we've had people I disagree with, we've had uh spirited debates, we've had random conversations uh going back over a decade at Beijan Conspiracy Podcast. Speaker 2 We argued with Simone and Malcolm Collins about whether death is good or bad. Which was fun. Speaker 3 Oh yeah. But we also had other sane things to talk about there, like population collapse and stuff, so you know. Speaker 2 So if you want to find that we are at the Bayesian Conspiracy. com, uh or you can look us up on YouTube, Bayesian Conspiracy, you know, uh all the places. If you look for that name, you'll find us. Speaker 3 Including any podcatch or podcast catching app. Speaker 2 Yep.