She doesn't particularly suspect that there's a Grimdark Conspiracy out to sabotage everyone else's ability to do science.  But only because Thellim has concluded that the Earth's moon is sabotaging fully everyone's ability to do science.

(In Thellim's defense, it is an extremely suspicious moon.  It grants magical powers during lunar eclipses.  Few dath ilani would not be suspicious that this moon had something to do with the problem.)

The problem with Bayesianism* is that it's too subjective.

(*)  A four-or-five-syllable Earth term that refers, if anything in their language does, to the forms for reasoning lawfully about probability as spotlighted by coherence theorems; though even among the few Earthlings who've heard of the word, far fewer have heard of coherence theorems.

Too subjective.

Too subjective.

Too subjective.

Would the Earthling care to explain what is less subjective about any of the deranged pseudo-mathematical constructs used in Earth 'science'?

Well, for example, let's say you flip a possibly biased coin 10 times.  It's an objective fact that if the coin is not biased, the long-run frequency of occasions where 10 coinflips have 9 or more heads, or 1 or fewer heads, is 22/1024.  So if we adopt the rule 'say the coin is biased if there's at most one head or at most one tail after 10 flips', we're adopting a rule which only rejects the null hypothesis, on occasions where the null hypothesis is in fact true, with 2% frequency in the long run.

This is a fact.  It is an objective fact.  Right?

It's a valid derivation from the stated assumptions, given the laws of probability, yes.  If you write a simulation program to flip fair coins ten times, and beep when the number of heads is 0, 1, 9, or 10, the program will beep with 2% probability each time it's run -

The more correct way to say it would be that the long-run frequency of occasions where the program beeps will tend towards 22/1024 as a limit.  That's an objective statement that is definitely true, where it's more controversial to say that it's meaningful to talk of there being a 2% probability of the program beeping on the next occasion it runs.  That either happens or it doesn't, so in what sense could the number 2% be true about it?

The way that truth-functionals on propositions generalize to truth-functionals on probability assignments, is that you say 2% and then you lose, like, 5 or 6 bits or so, if you're wrong, and lose a small fraction of a bit, if you're right.

It's slightly more complicated than mentally comparing the proposition 'snow is white' to a handful of snow and thinking that it's meaningful to talk of this proposition being 'true'.  Very slightly.  If you can do one you're probably smart enough to do the other, really.

If one denies that there is such a thing as a 2% probability that the program beeps on the next occasion, why couldn't you just imagine that you repeatedly run the program and it never beeps on any occasion forever?  That's a possible thing that happens when you keep flipping coins.  It's improbable but it's possible.

The probability tends to zero as a limit.

So this quantity that is supposedly meaningless tends to zero as a limit, which is, itself, apparently a very meaningful thing to say?  Yes, that sounds like a very consistent philosophy to have about the world.

After any finite number of steps, it will still, apparently, on his view, be objectively meaningless to talk about the probability that a program beeps at least once.

Therefore, the meta-proposition 'is it meaningful to talk about the probability of this program beeping at least once' will tend towards 'false' as a limit, on his philosophy -

Okay, look, it's not like that, it's more like - that the long-run frequency of programs beeping is 22/1024, is a valid consequence of the premise that the coin is fair.  That's just a true statement about the long-run frequency, with no probabilities involved.  It's how you reduce claims about probability to simpler claims that don't have probabilities in them, by looking at something which actually has a 22/1024 proportion of whatever.

This true statement about the hypothetical is derived from reasoning steps that go through probabilities.  After you've flipped the coin fifty times, for five runs of the program, there's thirty-two possible sequences of beeping and not beeping.  Extending these sequence lengths out towards an infinite limit, the sequences more than delta away from 22/1024 beep frequencies will tend to go under epsilon fraction of the total probability.  This is the only meaningful statement that can be made here about a 'long-run frequency', unless he wants to start taking direct ratios of countable infinities.  When the sequences are actually infinite, we can no longer directly speak of things having 22/1024 ratios inside them.

This limit is being obtained by considering the probability of the possible finite beep sequences, which is supposedly 'subjective' and forbidden.

Right, but the final conclusion is a certain one, which makes it objective, unlike the statement that a fair coin has a 2% probability of yielding less than 2 or more than 8 heads on the next occasion, when either it does or it doesn't.

Thellim might, under other circumstances, try to ask if his rules of metaphysics permit one to derive an 'objective' truth via a series of reasoning steps whose intermediates are 'not objective'.

She reminds herself that this person has in fact been driven crazy by the moon and she is not actually going to be able to argue him out of anything.  The main thing Thellim should do is note down that a manifestation of the compulsion appears to be a feeling that some complicated derivations of the probability axioms are 'objective', and the simpler steps of reasoning leading up to that conclusion are 'subjective', and that then the person refuses to accept the 'subjective' parts even though in principle they're how the 'objective' truth is derived.  They skip over the derivation, and feel like they can just go from the supposed premise to the 'objective' conclusion about long-runs, without any 'subjective' statements about short runs along the way.

Presumably this is what it feels like from the inside, after the moon has excluded from your mind all the simple and obvious ways to do science, and you've had to resort to fantastically complicated methods outside the boundary of what the moon is excluding.  The word 'subjective' is really just referring to whatever the moon prevents them from believing.  If you point out that their 'objective' statements are derived from 'subjective' intermediates in principle, they'll just jump all the way to the conclusion in what they claim to be one jump, maybe...?

Let's try a different tack.

Well, Thellim does in fact concede that the long-run frequency of occasions where this hypothetical fair-coin-flips-10-times program beeps, will tend towards 22/1024 as a limit.  She just thinks that it's not particularly plausible to have an epistemology of probability where you can say that, and exclude the idea that the next run of the program has a 2% probability of beeping.  In both cases you're just talking about a logical consequence of an uncertainty-containing model where the coin has an independent 50% probability of coming up heads each time, that your mind imagines being compared to reality using a logloss correspondence, and which you can actually score once you make the actual observations.

That this is not something you can derive from the long-run frequency of the coin yielding heads being 50%, as opposed to the statement that the coin has independently a 50% chance of showing heads on each occasion, is readily shown by considering the case of a coin that perfectly alternates heads and tails inside every run.  This coin has a long-run frequency of 50% heads, and also, the head-counting program run over it will never beep -

They're really getting a bit outside his area of expertise here, he's a scientist, not an epistemologist.  But presumably the definition of a long-run frequency also includes, for example, the idea that subsequences of two coinflips, HH, HT, TH, and TT, all have 25% frequency, within the long run.

This is either an infinite set of independent postulates none of which can be derived from each other or any simpler postulates, or they are the collection of simple deductive consequences of a model in which the coin has an independent 50% probability of coming up heads each time.

(Can he not see where his axioms are coming from, the excluded space inside his own mind... he can see the consequences of excluded beliefs, so they're probably still in there in some sense... it's just that when he thinks about them consciously, the moon forces him to label those thoughts as meaningless and 'subjective' and therefore bad, but that doesn't mean he can't intuit the consequences...)

Look, he's not actually a philosophy-of-science guy.

Being one of the smartest people on the planet, and the literally actually sanest person on it, is something that Thellim is still having trouble coming to grips with.  She isn't even a professional in any particular subspecialty of science, like report-writing or experiment-performance or experiment-analysis.  She just knows what eight-year-olds are supposed to know about science so they can read newspaper stories about it.

Which makes her all that this Earth has in the way of sane scientific methodology literally at all, and if she's going to do anything with that, she may need to get it done before the next lunar eclipse, which is now only three months away.

Can she possibly sell him on the idea that it would maybe be a better idea for experimental reports to report some different consequences of the probability axioms, even if those consequences are 'subjective'?  She realizes it probably feels very bad to talk about a consequence of the probability axioms that feels 'subjective'.  But, if it were super important, like maybe there were a lot of money at stake or something, people could talk about other deductive consequences of probability models that could be matched up against experimental realities?  Even though they seem to be 'subjective' and that's very bad?

Like what?

Likelihoods.  Likelihood functions.  Likelihood ratios.

If you flip a coin and see eight heads and two tails, you report, first of all, the actual exact sequence you observed including the ordering, and as summary, that the likelihood of this data given any underlying independent propensity p for the coin to yield heads, is equal to (p)^8 * (1-p)^2.

For 80% propensity it'd be... 2^26/10^10, and for 20% propensity it'd be 2^14/10^10, for example.

This is an axiomatic, deductive, fully valid and determined consequence of that hypothesis about the coin.

He knows what 'likelihoods' are, which makes him very smart and well-educated for an Earthling scientist!

He's heard that they have to be combined with 'priors' in order to make any use of them, though.  Priors are very subjective.

Okay, but... how much money would he need to be paid to report likelihood functions anyways, despite their being associated with priors, which are 'subjective'?  Is it possibly just a hundred bucks, if he focuses hard on the feeling, and considers how, even though 'subjectivity' seems like a very bad thing, there isn't any actual concrete terrible bad thing that happens to you?  Especially if you just report likelihood functions, and don't talk about any priors or posteriors on the real-world facts (as opposed to internal model variables being integrated out).  As is standard practice anyways, because other people might know something you don't, and because your report isn't a realtime-updated prediction market -

Well, no, the thing he'd worry about is that if you report on the data having whatever-it-was likelihood, given the coin's propensity to heads of 20%, somebody will decide that, even though this likelihood was pretty low, the coin still has a trillion-to-one prior probability of having a 20% propensity to heads!

Wouldn't this make the scientific process open to obvious abuses?  What then?

Ideally?  In a saner world?  Whoever says that can lose a ton of money on prediction markets about what the next experiment will show on the same coin!

Aren't prediction markets about that sort of thing illegal because of gambling laws?

Thellim is aware.  Sometimes she likes to pretend she's somewhere else where that isn't true.

To be clear, he's not especially a supporter of those laws himself -

Yes.  Thank you for the sentiment.

As it happens, a sane scientific process says that putting a prior probability like that, on any particular effect-size, or reporting a posterior after seeing the data, isn't supposed to be the job of experimentalists in the first place. So maybe this is a place where that whole horror of the 'subjective' could work in their favor.

And has he noticed that there's... kind of a lot of end-user-manipulable free variables inside the 'frequentist' procedures that have been constructed in incredibly elaborate ways so as to avoid all mention of likelihoods?

...is she talking about p-hacking?

She's talking about how, if you flip a coin and get the sequence HHHHHT, then this is 'statistically significant' if you got the sequence by following the rule 'Flip until you get the first tail and then stop', in which case the sequence is only six coinflips or longer on 1/32 of occasions, p < 0.05.

And 'fails' to achieve statistical significance if you got exactly the same exact observations by following the rule 'Flip the coin six times and count the tails', in which case there's 7/64 ways to get one or fewer tails, p=11%, n.s.

Well, no reputable journal would accept the first analysis - it's not the way somebody would normally compute the p-values after collecting the data on six coinflips.  Which isn't really very much in the way of coinflips anyways -

Or take 95% confidence intervals.  One way to get a 95% confidence interval is... whatever the traditional method is, Thellim tried to read it several times but her eyes started to bleed and black tentacles started squirming out from under nearby doors, but it involved taking the average of all the measurements so far, and picking a data-dependent constant c to add and subtract to the mean, to get an interval (mean - c, mean + c) such that following this procedure would, 95% of the time, in the long run, give you a confidence interval that contained the true population average value of the parameter, if you were sampling randomly from that population.

There's also, obviously, going to be a way to pick a constant d where you report the interval (mean - d, mean + 2d) via some method for picking d such that this method will 95% of the time, in the long run, produce an interval containing the true population mean.

Yet another way to get a 95% confidence interval is to, like, construct a 99% confidence interval the traditional way, and then with 4/99 probability, report the interval (purple, blagoobah).  It doesn't matter that blagoobah isn't a word.  The interval will still contain the true average 95% of the time, in the long run, if the method is separately repeated infinitely many times.

He agrees that the second and third methods are worse ways to get 95% confidence intervals than the first method; they'll produce wider intervals with no gain in correctness.

What she's trying to get at here is that all the 'p-values' and 'confidence intervals' are much more manipulable than likelihood functions as a summary of the data!

P-values contain free parameters for which class of other possible results you decide to lump in with the exact data that you saw.  If you say that the result HHHHHT is part of a class of results that includes {THHHHH, HTHHHH, HHTHHHH, HHHTHH, HHHHTH, HHHHHT, HHHHHH}, then there's 7 lumped-in results like that with total probability 7/64, so you say 'not significant'.  If instead you say that HHHHHT is part of a class of results that includes {HHHHHT, HHHHHHT, HHHHHHHT...} then there's infinity lumped-in results like that with total probability 1/32, so you say 'significant'.

Similarly with 'confidence intervals', and all the different ways you could fiddle interval construction, and still have it be a valid deduction that the method would with 95% probability - pardon her, would with 95% frequency in the long run - produce an interval containing the true average measurement.

Likelihood functions don't have free parameters like that!  You don't take the actual results you saw and lump them in with other results you didn't see and calculate their summed probability!  You don't apply 'methods' to things to draw weird intervals around them!  You just use the actual data that you saw!  For any sufficiently characterized way the world can be, any hypothesis of a sort that experimenters ought to summarize statistics about, there's just some valid deduction about how likely that world was to produce the exact data observed.

You can't fiddle with the numbers if you're honest, is the idea in science.  Science does presume that scientists were honest in reporting which rules they followed to do their experiments; it does assume that somebody who flips a coin six times, gets HHHHHT, and stops, will honestly say that 'they decided to flip the coin six times and then stop' and not lie and say 'I decided in advance that I'd flip the coin until I got tails and then stop'.

So long as nobody lies, though, the free parameters can't be used to cheat - you can't have some clever way of picking the method that lets you get p < 0.05, with probability greater than 5%, when the null hypothesis is actually true.

Science does presume honesty; but absent that presumption, people could just lie about the data anyways, no matter what stats you used.

When you flip a coin and get HHHHHT, the meaning of that result should not depend on the experimenter's state of mind unless the experimenter's state of mind is able to affect the coin.  What tells you about reality, is what is entangled with reality; the coinflips entangled with the coin; if the experimenter is not telekinetic then who cares what they were thinking.

Once you have characterized the world, the likelihood of the data, given that world, doesn't change with what the experimenter is thinking, unless the model says that the experimenter's thoughts are able to affect reality.

And let's be frank here, if Earthlings are not entirely immune to temptation in the face of terrible incentives and no prediction markets and not very much replication, people might be quantitatively more tempted to fudge their unobservable private intentions while conducting their experiment than to fudge the hard objective facts of what was observed that somebody would be more likely to catch them lying about.

(Thellim would ask how an epistemology with experimenter's-unobservable-private-intentions-dependent interpretation of the evidence is not 'subjective', but she has worked out by this point that 'subjective' actually means 'the moon makes me feel like I hate this' and doesn't really relate to the ordinary English meaning of the word.)

Well, that's why there's very standard methods for computing the p-values, which, in practice, gets rid of a lot of the downsides of the 'free parameters' Thellim is complaining might exist in principle.  Nobody's actually going to believe you if you flip a coin six times, get HHHHHT, and claim you decided to flip until you got a tails and then stop.

With very very simple statistics, that might, possibly, be true with respect to that exact particular class of disaster that results from 'p-values' lumping different hypotheses and outcomes into weird buckets.  There's other disaster classes she'll get to.

But Earth science has been known to ever involve complicated procedures involving dozens of parameters entered into 'stats' computer programs.

Past that point, if the reported-on strength of your evidence isn't independent of the experimenter's private state of mind - and there's no prediction markets, and there's no preregistration of studies including the analysis methods they'll use, and journals accept papers after the evidence gets gathered and analyzed instead of accepting the preregistered paper beforehand, and the journals openly take into account the results when deciding whether to publish the study, meaning experimenters are being openly plied with incentives that are not accuracy incentives - you're kind of screwed.

Well, he's definitely not arguing against the point that there ought to be more preregistration of studies, though the idea that journals should accept papers based on their preregistration is one that he hasn't heard before.  He likes it in principle, but it's the sort of noble idea that basically never happens in real life.  His field is still trying to replace Elsevier journals with open-access ones.

Ah, yes.  Elsevier.

Earth 'science' is a giant obvious flaming disaster, and you'd think people would be more open to the suggestion that maybe the incredibly complicated private-state-of-mind-dependent statistical analyses would perhaps also be broken and contributing to the giant flaming disaster.

He really doesn't think that bad stats are the primary problem there -

Every single aspect of Earth's science is broken SIMULTANEOUSLY and the stats are PART OF WHAT'S BROKEN.

The 'null hypothesis significance testing' paradigm means that a 10% apparent effect size on a small group and a 3% apparent effect size on a larger group can be treated as a replication having 'reproduced' the original's 'rejection of the null hypothesis' instead of as a failure to reproduce the apparent 10% effect size.

The 'null hypothesis significance testing' paradigm means that journals treat some findings of an experiment as 'failures' to find a 'statistically significant effect' rather than as valid evidence ruling out some possible effect sizes.  That journals then don't publish that evidence against larger effect sizes, because they didn't accept on the basis of preregistration, is an enormous blatant filter on the presented evidence which no sane society would tolerate for thirty seconds, and also, a giant blatant incentive that is not an accuracy incentive.  If you think in likelihood functions there are no failures or successes, there is no 'significant' or 'insignificant' evidence, there is just the data and the summaries of how likely that data is given different states of the world.

If you've correctly disentangled your hypotheses, likelihood functions from different experiments just stack.  You literally just fucking multiply them together to get the combined update.  It becomes enormously easier to accumulate evidence across multiple experiments -

- although, yes, anybody who tries this on Earth will no doubt find that their likelihood function ends up zero-everywhere, because different experiments were done under different conditions, as is itself a vastly important fact that needs to be explicitly accounted-for, and which the "rejection of the null hypothesis" and "meta-analysis" paradigms are overwhelmingly failing in practice to turn up before it's too late.

'Null hypothesis significance testing' rejects the notion of a way reality can be that produces your data.  It just says reality isn't like the null hypothesis, now you win, you can publish a paper.  That doesn't exactly prompt people to notice if reality is being unlike the null hypothesis in incompatible ways on different occasions.

The 'p-value' paradigm's dependence on the experimenter's private state of mind means that people can't just go out and gather more data, when it turns out they didn't get enough data, because the fact that the experimenter has privately internally chosen how much data to gather breaks the p-value paradigm -

Wait, is she saying that people should just be allowed to gather more data any time they feel like it?

YES!  It's just DATA!  It's not going to HURT YOU if you don't MISTREAT it like Earth scientists do!

If the experimentalist's intentions are not telekinetically affecting the data, then your statistical method shouldn't care why they decided to gather the data, the data is telling you about the world and not about the experimenter!  People can gather data for any reasons they feel like!  Or no reasons at all!  Stop telling experimenters how to feel about their data!

Okay, but then, what stops somebody from just continuing to flip a fair coin until there's a bunch more heads than tails in the results, and then stopping and reporting that the coin is biased towards heads?

He is welcome to set up any computer program he likes, in which some coins are biased, some coins are fair, as selected at random according to a known distribution; and an analyst, who knows this prior distribution, is updating her credences correctly using likelihoods; and an experimenter, who does not know the true state of the coin but can observe it, can decide as he pleases when to stop gathering data.

The experimenter will not be able to make the analyst arrive at an ill-calibrated probability distribution.

Now, maybe if the analyst had bad priors, they'd come to bad conclusions.  Which is why experimental reports shouldn't report on posteriors like that.  But the likelihood functions don't introduce any problems for the analyst, even if the experimenter is deciding how much data to gather based on previous observations; because the experimenter's private decision process does not then affect the likelihood of the additionally gathered data given the true state of the world.

...he doesn't see how that prevents, in practice, the experimenter from continuing to flip a fair coin until it shows an excess of heads.

He might perhaps find it illuminating to write some computer simulations along these lines, until he has been convinced of the point that you cannot, in fact, convince a lawful reasoner of anything you want, by deciding when to gather more data.

After all:  If that were possible, a lawful reasoner could plan to convince themselves of a fair coin being biased heads, by deciding themselves to keep flipping until it showed an excess of heads.  And then they would know in advance which direction they would update later, as is obviously mathematically impossible.

Huh?

It's magic math.  Earthlings will never be able to understand it.  It comes from another world and has inscrutable mystical properties like "If you start out thinking it's 70% probable that some coin is fair, nothing you can possibly plan to do by gathering more data, analyzing it a particular way, etcetera etcetera, can result in you expecting for that analysis to make you believe on average that the coin is not 70% probably fair."  All of the sinister misdeeds that 'p-values' are trying to prevent by, for example, not letting naughty experimenters gather more data whenever they feel like it, are just flatly mathematically impossible in the first place under this system.

What she's hoping at this point is that she can get a system set up where some Earthlings are using the magic math, even though it seems very 'subjective' and that makes for weird internal feelings that make them want to reject it, paying Earthlings enough to do it anyways, just, just trying it to see if it works, and maybe leads into less massive replication failures, and then they get interesting results that other people want to pay for, and the system can continue under its own momentum.  And prediction markets; and professional replicators; and impact markets instead of journal citation counts, for important results that really do reproduce, to establish purer accuracy incentives; and specialized analysts and specialized writers; and everything else the system needs to fix everything broken about it simultaneously, in one small part of it somewhere.

Thellim has three months left, on one of her dominant hypotheses about how this world works, before she suddenly starts thinking that likelihood functions are 'subjective' and meaningless and icky.  She has to build something that will sustain itself before then.

Yeah, nothing's getting built in three months, he's pretty sure.

Why not?  Why?  Three months is so much time!  It's 90 days!  People can do a lot in 90 days!  They just all have to make the correct moves simultaneously!  It's not that hard to coordinate!  They just have to look inside them for whatever makes them feel icky about the thought of coordinating and ignore that icky feeling, just once, just this one time, and they can maybe set up a fragment of real Civilization that would - that would, even if something happens to Thellim, be able to figure things out, put her back, eventually, to the way that she was -

He has to go.

Thellim is aware that Earth is not literally pessimal.  But sometimes it's hard to see how her situation could realistically be worse, anthropically speaking, assuming it's a requirement that she landed in a populated world instead of somewhere in intergalactic space.  It feels like this planetary setup is about as bad as a planetary setup could get without stuff actually collapsing.

...why couldn't she have just ended up in a regular fantasy universe, one where she'd have her own economicmagic already, and without her mind having been warped in the process?  Someplace that would actually listen to her, and let her improve it.

Maybe with a harem.  A harem of magical boyfriends would be nice.

She hopes the other dath ilani on the airplane ended up in better places.

Somebody has to be worst-off, after all.  With any luck, that was her, and the other dath ilani are in happier, less fraught, less dangerous situations than this.

(Return to the main Planecrash continuity.)