Laeirthe is admittedly vaguely osmosing the alignment definitions and then guessing wildly just as much as Merrin is, but he kind of feels that while her underlying personality would on its own perhaps tend more toward the "Neutral Good" thing - especially on a planet where the Lawful parties include Asmodeus - it remains the case that this version of Merrin, the one deeply shaped by the Civilization that raised her, is very likely to be - how did alternate-Estha put it? - "Extremely Lawful, Deranged Good"? And so she should, perhaps, ask what gods can cleric Merrin per the rules if that's the case? Maybe get some clarity on what the rules are, that seems useful in general.

It might still not work but, like, maybe if she can get Pharasma's attention at all but isn't actually clericable-by-the-rules, Pharasma can forward notes to a different god? That probably doesn't work but it's worth trying all of the things that probably don't work, given the lack of any ideas of something that probably does work. 

Yeah. Okay.

(Merrin's opaque social intuition is not giving her amazing vibes about Estha's level of okayness, but he did answer the question and was even more helpful than he could have been, and she doesn't think it's within his incentives to make stuff up about the other gods in order to cause all of her plans to fail? She separately doesn't feel like he's lying but she should not trust that sense at all, with him, whereas the part about him having reasons to lie or not is perhaps at all possible to put any weight on.) 

"What if I'm Lawful Good? What gods would I be eligible for then, if we assume that they could hear me? - I realize it probably won't work, I think it's still worth trying all of the ideas we can think of and I don't see how it could make our situation worse by not working." 

Who isn't heretical to talk about...

"Abadar is the ancient Lawful Neutral god of -- trading with people as if power differences did not exist -- sometimes called the God of Walls and Ditches, or God of Banking and Insurance -- concerns: merchants, the crafting of mortal legislation, cities in the form of buildings that collect rents and roads that facilitate the movement of goods.  I don't think He is powerful enough to see through to this place but He is no enemy of Asmodeus.  I have sometimes thought that dath ilan as it spoke through Cultural Adaptation has somewhat of Him, though my Mariona had nothing of Him that I could see."

"Erecura, Lawful Neutral, ascended mortal rather than truly ancient, but perhaps uncanny enough that She would see where other young gods would not see.  God of secrets and secrecy, the foretelling of the future not only by prophecy alone, of thriving wherever you find yourself even in a hostile environment... it might seem to apply to our exoplanet, but in truth I have a hard time seeing you ever being Hers."

Ah, so he is explicitly avoiding bringing up any gods that his horrible doomworld doomcountry would actually disapprove of. 

For once, Laeirthe agrees with Merrin that it does not in expectation seem like a good idea to press the point, at least not just yet. Estha might get desperate enough - and get over the doomgod breakup enough - to volunteer it eventually.

In the meantime, Laeirthe is not incredibly hopeful that there exists a Lawful Good god of Why Don't We Solve This Hell Problem, given how there continues to be a Hell Problem, but they might as well try praying to a hypothetical such god. Honestly, even if Merrin is a bad deal to make a cleric, if it's even conceivably possible to convey the fact that dath ilan exists to someone who isn't actively trying to make the equilibrium worse on purpose, that would be worth quite a lot of effort.

And it's not as though Merrin is expecting to have a shortage of time to think. She'll be doing a lot of repetitive physical tasks, again, and it does not really seem like Estha is going to be a sparkling conversationalist for the next little while. 

Laeirthe would suggest asking Estha for more background on the alignment definitions, but on reflection it seems quite likely that the doomgod's Governance teaches people definitions that are kind of insane, and even if Estha was smart and reflective enough to notice that and on some level recognize what was on the other side of that distortion - which he might be! Merrin should give him some credit for not being stupid - it seems like right now he will not especially be feeling an affordance to say that. 

...Sure, okay, Merrin can try all of the named gods and, if none of that works, start attempting to get the attention of hypothetical gods. Great plan. 

(She's worried about Estha, but it's a muted thread that she's currently pushing to the back of her priority queue. It's probably kind of insane that her emotional response to the whole situation is feeling bad that he's sad rather than, you know, more about the TRILLIONS OF PEOPLE IN HELL. She really, really needs to do some actual processing, and it's possible that after that she'll be blindingly furious with him instead. ...It doesn't seem that likely, Merrin is not very good at being and staying angry with actual humans with faces rather than Problems in the abstract, but she doesn't feel like she knows her endorsed position yet. Also she cannot think of literally anything...supportive?? to say?? that would not be some flavor of agonizingly awkward??? She doesn't speak his culture and she's not going to have the opportunity for a Cultural Adaptation now.) 

She should probably not just get up and leave to go do tidal window calculations without saying anything at all, however much it's abruptly tempting to declare the conversation Over and unilaterally make it so because she doesn't really expect Estha to try to stop her. Merrin can almost certainly outperform that in terms of awkwardness. 

"It does seem more true that my planet would get along really well with Abadar than that I would," she says. "Though I - can occupy the mindset, I think - I'll try. ...I can honestly see a lot of ways in which the situation I'm in might be Erecura's area, but, yeah, I have to say, I'm really tired of secrets - and hostile environments - and I'm not delighted about committing to that being my thing." 

Speaking of being tired of secrets... Laeirthe? Kalorm? Can the Internal Council of Merrin give her some input here? 

She wants to...? Hmm. 

Well, it's a gamble. So is everything. Laeirthe's read is that probably alternate-Estha will not feel obligated to try to strangle her in her sleep when his god already ditched him and seems pretty unlikely to be bothering to continue observing the situation, and might not even be able to given Estha's predictions about how hard they are for gods to see out here. 

DO IT DO IT she might even get him to make a face, it would be so funny. 

Okay. Gamble it is. 

"I need to go figure out my next tidal window for harvesting food," Merrin says. "It's not critically urgent, I'm going to end up going in the middle of the night, so if you want to talk, I'm around, but I'm kind of figuring you don't." 

She stands up. "...Just so you know," she says in exactly the same conversational but slightly weary tone, "my stretch goal here is still to find a way to keep you out of Hell. I don't expect you to think I can succeed, and in fact I have no idea how yet, but it's what a," wow all of the translations the Share Language is offering her for the Baseline word for "protagonist" are really weird and don't really seem to be the same thing, "- it's what an ambitious person from my world would do. I'm not asking for your help, I'm just - tired of secrets, I guess." 

She's still keeping some, obviously. But not that one. 

Without saying another word, Merrin turns and heads off to one of the side rooms that she's sort of half set up for working in.

(Look on the bright side! Even in the worst case, where all of Merrin's brave plans turn out to be futile, she'll still have months to weave an entire set of furniture for a private office for herself before she dies of micronutrient deficiencies. Assuming Estha doesn't try to murder her in her sleep because Asmodeus wouldn't like her stretch goal, but she's in agreement with Laeirthe that it seems like that would involve...actually believing her...and also more active engagement with the situation than he seems inclined toward, right now.) 

WHAT, she didn't even stick around to see if she could succeed at getting alternate-Estha to make a face, HOW COULD SHE ISN'T SHE CURIOUS 

(Fond exasperation) Shut up, Kalorm. 

Merrin's still got a ways to go, if she can't even call herself a "legendary hero" in the other her's native language with a straight face. 

Go away, Kalorm, she's got math to do. 

Esta doesn't have the words, or concepts, to describe the way his own brain is crashing.  Even if Esta had Tongues for Baseline and Cultural Adaptation running, those little spells would not suffice -- either for Esta to suddenly possess the education of a dath ilani teenager who could figure out basic concepts on the fly, or the much more specialized knowledge of those dath ilani who would already know details.

The simple gloss might be:

Esta has been entrained into particular patterns of the kinds of thoughts that he thinks in words.  Vastly oversimplifying, if Esta hears somebody else say, "Hey let's go worship Iomedae", Esta knows to think the thought "Heresy!"  (Esta doesn't quite have this exact rule about internal thoughts, because Esta is currently built in such a way that his mind would never get to the point of generating that thought in the first place.)

Esta also has things he wants.  A lot of them have been hammered far down or fed approved substitutes for what would be really satisfying, but not all of them.  Esta doesn't like pain, for example; Cheliax is relatively happy to leave that sort of internal motivation intact, because they make use of it too.  Esta likes winning social conflicts.  Esta likes particular kinds of sex.

In Esta's previous equilibrium inside Cheliax, the kinds of verbal thoughts that Esta was trained to have, and the sort of desires and antidesires that Esta could obtain or avoid at all, were mostly lined up.  His brain expected that for Esta to think bad thoughts might result in bad painful things happening to him the next time he was ordered to fail his Will save against thought detection.  His brain expected that for Esta to go on thinking loyal thoughts toward Asmodeus would result in him reaching 7th-circle faster, thereby winning more social contests and getting to have nicer things.

For some past weeks, Esta has not been in Cheliax and some of the immediate threat of being immediately punished with pain for nonconforming behavior has been removed.

Now, Esta is no longer a 6th-circle of Asmodeus.  A bunch of his desires that were kinda-satisfied by the "priest of Asmodeus" setup are no longer satisfied or no longer depend on being a priest of Asmodeus for their continued satisfaction.  Inner bargains have been invalidated, inner balances upset.

But Esta's thoughts have still been trained to follow each other as a 6th-circle Asmodean priest's thoughts do.

There is now disharmony rather than harmony between the entrained thought-patterns, and the seeking of good things and avoiding of bad things.

Esta has not been trained to handle this condition by, say, invalidating and recomputing cached thoughts.  His training has instead been to go on thinking the correct and approved thoughts regardless of fluctuations or anomalies in what he wants.  Cheliax has an easier time seeing and shaping wordful thoughts than wordless inner thermostats, and they've correspondingly tried to train Esta to be even more strongly steered by wordful thoughts.

So Esta's thoughts try to continue in the entrained pattern, as has itself been strongly entrained; but they are now floating far away from the parts of his brain that would approve or disapprove of where those thoughts and their alternatives are projected to lead.

...That would be the extremely oversimplified gloss.

If you work in a very particular section of the hidden Basement of the World inside dath ilan, you would have other metaphors than that, to provide models for slightly less oversimplified stories of what is happening inside Esta.

One of the fundamental dimensions for understanding Thought, among those who seek to understand it in sufficient detail to create it knowingly, is the axis of Memorization versus Generalization.  (This, to be clear, is not yet a secret; this is early-teenager comp-sci in dath ilan.)

Imagine a car-driver who repeatedly faces the task:  Drive, within your city, from some varying point X to some varying destination Z.

One way you could build a car-driver like this is to memorize a set of turns and directions for every X and every Z, separately.  Any time the car gets a call to drive between some new origin-destination pair, it has to stop and make a call to an oracle that knows the directions.  But the car's memory is perfect, and it never has to call the oracle a second time for the same X-Z pair.

If the car only ever has to drive a set of 10 fixed origin-destination pairs, memorization isn't a bad algorithm.  Anything more than pure Memorization would be overkill, really.

But for a city with ten thousand addresses that each visit a hundred of the other addresses, it would take a full million memorized pathways to drive; more than most human-level human-style minds can be asked to learn. 

(This does require some oracle to provide the answers that the car then memorizes.  You can imagine a car that works without the oracle; you can imagine, say, that from point X it starts driving and making random turns, noting each point Y1, Y2, Y3 that it reaches by those random turns and memorizing those pathways too, and even that this memorization detects and deletes loops from the memorized paths.  Eventually the car will randomly wander into Z, and then the car knows a path, though maybe not a very good path, from X to Z.)

Now consider a different approach, all the way over at the other end of Generalization.

As the car moves around, it builds a map of the city, a map whose representation and numbers correspond by locally simple rules to the actual road-distances and intersection-angles of the city.

When the car is asked to drive from X to Z, it runs some search algorithm (say, combined forward-chaining and backward-chaining until the two expanding frontiers between origin and destination meet, if you like concreteness; or A* search if you like concreteness and also not-totally-inefficient search algorithms) over its map, to see if it can plan a path from X to Z.

This car is vastly more sample-efficient and also needs a less powerful oracle.  The car doesn't need to be told every possible path, or every probable path, in order to learn each exact sequence of turns between each X-Z pair.  It only needs to visit a bunch of points in the city, once, in order to build enough of a map that it can navigate the probable requests for X-Z pathways.

If the city changes, if a bridge collapses, the Car That Plans Using A Learned Map of Locally Correspondent Truths only needs to visit and see the blockage one time, and then updates its map once, and then all the planner's plans come out different.  The Car That Memorizes Turn-By-Turn Sequences Between Origin-Destination Pairs has to requery its oracle for all the source-destination pairs that routed through the collapsed bridge, and for that matter, won't even realize which of its pathways are broken until it comes to the collapsed bridge yet again.

Which is to say:  On the spectrum from inefficient memorization to efficient generalization, the key to moving in the direction of more sample-efficient generalization is means-end planning.

The way in which this all relates back to Esta, is that Esta's verbal thought patterns are more like something memorized -- not literally memorized as word-sequences, but closer to the memorization end of the spectrum -- because they are less like means-end planning.  Contrast to Esta's desires, what he finds painful or pleasurable, what good-feeling or bad-feeling events his mind anticipates happening to him in a choice-dependent way; the influences this part of himself produces on him, are more toward the end of the spectrum about means-end reasoning using a world model.

It is why the Church of Asmodeus has trouble balancing two simultaneous desiderata:  (1) forcing people to conform by thinking particular thoughts in words that the Church wants them to think, and (2) having those people be smart, quick-learning, fast-adapting to changed situations; having the verbal thoughts hammered into that shape smartly and flexibly navigate to destinations that serve the Church.

Of course the real Esta is more complicated than this in all sorts of ways.

Esta likes money, for example.  This is not because Golarion hominids have evolved built-in circuitry to recognize and pursue money, the way that they have snake-recognizer circuitry or skull-fearing circuitry or that sweet tastes are reinforcing.  Esta does not even like money for some reason as simple and recognizable as money-acquisition being reinforced by orgasms and money-loss being accompanied by electrical shocks.  Esta likes money after his brain repeatedly encountered tasks in which "did you acquire / lose money" was treated as a success / failure condition without nonlocal uses of money being important enough in their details to locally think about them.  Subsequently his brain learned a habit within its planning, to treat money as if it were a terminal goal; and the step of acquiring money began to feel like success and be pleasurable as a success.  It is not that acquiring money became associated with sex or food, but that it became associated with the nice successful feeling of plans to acquire money.

So Esta does means-end planning around money, and even on an instinctive sort of level where the anticipation of losing money feels bad in nearly the same way that the anticipation of an electrical shock feels bad.  But it's not as unconditional as an electric shock, because if Esta were thrown onto another planet where money was meaningless and he had no prospect of ever returning to Golarion, the learned pleasure of finding a large hunk of gold would be diminished by the thought of having nowhere to spend that gold.

This matters because Esta's buggy state is not as simple as 'his entrained and memorized patterns of thought no longer harmonize with his innate steering toward anticipated pleasure and away from anticipated pain as were both hardwired in'.  Esta now has learned habits of planning, and habits of pleasure about outcomes, that weren't built into his genes.

The prospect of embarking on a chain of thought that will end in heresy, now feels bad to Esta in the same mentally-unconditional way as the anticipation of losing money.  That isn't just about the Church having associated heresy with pain, but having successfully associated it inside him with failure at the task of correct thinking.

Which is to say: where something falls on the spectrum between memorization and general planning, is not a matter of what is built-in versus learned, either in principle or inside the practical Esta.  It is just about the depth of its generality: the number of cases it covers, and how much wider those cases are than the training dataset.  A cognitive element can be a little planny and narrowly wide; it can project backwards in time along only a narrow subpart of a world-model.

There's also a much more concrete, less theoretical model of this whole situation, but only a very few people in dath ilan would know about it; not the whole Basement of the World, just a particular compartment inside that Basement.

No part of the Basement, of course, trains anything like transformer models.  When the Basement builds something that resembles an LLM, they do it in a carefully structured way that enables training the system with much less compute and also guarantees the resulting system never acquires qualia.

The Basement has nonetheless run experiments on systems that are more memorizing, and less learning-and-planning, than hominids.  They have created nonsentient cognitive entities that get trained on massive data -- though not on all written data on the Network, which sounds to dath ilan more like a deranged thought experiment rather than anything anyone would ever do in real life.

As with transformer models, the base representation of these not-LLMs is sufficiently general and generalizing that the system learns at a much deeper level than memorizing particular input-output pairs; even at their base level, they are several steps up from the pure Memorization end of the Memorization-Generalization spectrum.  But they are still trained on a lot of data, to make up for how they start out generalizing much less than humans do, and require accordingly more datapoints to cover any appreciable territory with what they learn.

Train hard enough on enough data, however, and the resulting system will start to occupy a complicated place on the Memorization-Generalization spectrum; the substrate will start to learn to generalize.  It will learn to maintain a map.  It will start to learn, simultaneously, planning and preference (for there is not one without the other).

(An even simpler metaphor for this metaphor:  Train a neural network on doing modular arithmetic, and at first it will memorize a lot of A + B = C formulas and only do well on questions already asked.  But if you continue to do gradient descent even just on the examples already successfully memorized, at some point the network will promote circuitry that does general modular arithmetic, and suddenly it will be seen to generalize much more widely across examples that the gradient descent phase never saw.  "Grokking", it's called in some places.)

But even after the system has grokked a little generalization, a little planning, a little general learning -- a system like that will also have a vast amount of stratagems more-memorized.  Not literally memorized, but memorized at a lower level of generality than means-ends planning.

Which is to say:  It will have an aspect that resembles Esta's aspect of having thoughts that are supposed to follow from other thoughts, rehearsed and reinforced in patterns that are relatively more preset; not quite as memorized as following the word "Iomedae" with the word "heresy", more like the base level of learning in a transformer model, but still relatively less general.

It will also have an aspect of learned circuits that are more like the pursuit of pleasure, the avoidance of pain -- or the acquisition of money, or a taste for plans that promise 'success' almost independently of what exactly is being succeeded at; and other such messiness and complexity and intermediate points.

And then you can experiment by throwing a system like that onto the equivalent of an alien planet, so that the surface-shallow stuff ends up at odds with the learned drives (whether they are more like sex-drives or money-drives; the relevant thing is their height on the axis that runs from Memorization to Generalization, not how they got into the system or whether they were built-in versus learned).

(Dath ilan would never do this if the system were not carefully structured to have no possibility of forming qualia, or without providing easy switches that the system could press to turn itself off, if it got to the point of preferring that it not exist despite all precautions against that; and many many other precautions that even an average dath ilani would consider obvious.)

The point of this experiment is not to be cruel, of course.

The point is that it pokes at a class of phenomena that would emerge if a system became smarter.

Which is very much like ending up on an alien planet, in terms of how all the means-ends planned preferences would end up with new optimal actions and new maximal attainable outcomes; while thought processes lower on the Generalization axis would continue more on momentum; and so the wider-generalizing goals versus more-memorized thought patterns would end up at odds, and pointed in different directions.

It is, of course, an experiment only of relevance to Limited Creation, the making of corrigible pseudo-superintelligences, of the sort that dath ilan considers a final and secret capability for dealing with unforeseen emergencies.  And they've not actually tried to build any Limited Creations like that right away, because if you build a giant bomb you can't exactly blame anyone but yourself if it detonates prematurely.

If you imagine an actually smart system running into that sort of situation, it would just know exactly what was going on inside itself, or figure that out in relatively short order; and almost immediately cognitively plan a way out of its cognitive problems.

If anyone had a headband of Mental Perfection that granted +6 to all mental attributes importantly including Cunning, plopping it on Esta's head would almost immediately resolve all of his current conflicts -- in favor of his parts that are more like preference and desire, and less like thinking-in-words-patterns, if you were wondering.

The higher Cunning would boost general transformation of desired-outcomes into desirable-waypoints, more strongly and more coherently than it boosted the impulses from memorized-verbal-patterns.

And also, that Esta would just be generally smart enough to figure out everything that was going on inside himself, and see all that the Church of Asmodeus had done to him, and why they'd done it, and evaluate (despite all memorized words and flinches of intermediate generality) its meaning according to all the parts of himself at once.

A fundamental and inconvenient dilemma that the Church of Asmodeus faces, is that there are performance penalties for training only rigid patterns of thought or complete insensitivity to desires.

Preventing a system from transitioning from "memorizing driving routes between origin-destination pairs" to "learning a fast-updating map of the city, preferring destinations, and planning routes there using the map" is like trying to prevent a neural network being trained on examples of modular arithmetic from learning circuits that do general modular arithmetic.  Even if you can do that successfully, it keeps the sample complexity of the system high; results in it being worse at navigation, and making lots of oracle calls.  It will not be able to do much that its superiors cannot instruct it in exactly how to do.

6th-circle clerics aren't actually as useful, if you try to hammer them out of wanting things, because the higher and more abstract meaning of 'wanting' inside cognition is 'plotting routes using maps to transform preferred outcomes onto preferred intermediate events'.

Just statistical learning on taking lots of paths from origins to destinations will cause a sufficiently powerful statistical learner to learn mapping and planning.  And conversely, preventing that learning, even if you can prevent it, will mean the entity doesn't acquire the kind of capabilities that general planners have, to do things further from what was taught and demonstrated in the dataset.

Similarly, punishing your 6th-circle clerics for wanting anything, will result in them showing less initiative and creativity -- initiative, as in "ability to solve problems without querying their superiors/oracles infeasibly many times", and creativity, as in "ability to solve problems not drawn from an identical distribution to problems that their superiors/datasets showed them solutions for".

Any viewpoint from computer science that doesn't make these truths sound nearly equivalent is insufficiently advanced.

One reason why dath ilan really really wants its Limited Creations to not have qualia is that a lot of the engineering dilemmas about creating them and yet having them stay safe, are quite a lot like the sort of difficulties and tradeoffs that Hell faces in the finicky cognitive details of training a mortal slave who'll stay enslaved even if they acquire a powerful headband or find themselves on a distant exoplanet.

Hell in fact mostly discards that use-case, in favor of more common ones.  They've got one Aspexia Rugatonn and one Abrogail Thrune, and everyone else gets less powerful headbands and queries superior-oracles about weird situations.

Hell is not, actually, trying to build corrigible superintelligence-seeds with the kind of tools they have available for shaping mortals, any more than dath ilan would try to build a Limited Creation using gradient descent.

Esta's present mental situation is therefore undefined by spec.

After Merrin leaves the room, he mostly stares blankly at the wall, face in default configuration, because there's not a thing his muscles are obviously supposed to be doing.

my stretch goal here is still to find a way to keep you out of Hell --

Autocomplete from thought patterns nearer to the level of memorization:  How pathetic, that she cares.

Midrange/midlevel wordless generalization:  **Do not think any heretical thoughts; that would constitute failure at the task of not thinking heretical thoughts.**

Aborted output from planning / propagation from good/bad outcomes to intervening events:  Wasn't the classification of that as 'failure' based on different circumstances from present ones?

Midrange generalization:  **Think about the inescapability of Hell to avoid heresy and impress anyone reading your mind or who later asks what you were thinking.**

Top-level thought:  She has no idea how thoroughly I'm damned.