Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness.
One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.
After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.
If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.
Since we all know that at this point, someone is definitely going to build the torment nexus, we all may as well get involved and help, so that at least we can exert some limited degree of control over the eventual character of the torment nexus. At least that way maybe we can make it a little less terrible. At any rate, it can’t make it worse than it would have been if we -didn’t- help build it!
I've been working with self improvement harness a little bit and one thing i've come to conclusion is harness task fit. The learning can be significantly improved if we understand the behaviour of task and how it should be learned.
I'm pretty sure a general solution will definitely exist which will do fine, but we are yet to see one.
Agree with the sibling comment that an example would be useful. I'm also curious what kinds of tasks you have in mind. Theoretically almost every knowledge work task boils down to the file editing that agent harnesses already are built around (which probably explains the second half of your comment).
one form of very effective self-improvement that coding agents do all the time:
install or build stuff that they can then use
it changes the environment instead of the agent/harness but in a sense how separate is the agent from its environment and why do we apply this distinction re self-improvement?
animals and humans do the same thing and are great at it, without 'self-improvement' with emphasis on the 'self'
Blame Descartes? I don't know. Extended theories of cognition all get shot down, even analytic philosophers who are accused of being more "continental" still primarily hold mind-body dualisms
One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.
After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.
Can you make an example?
install or build stuff that they can then use
it changes the environment instead of the agent/harness but in a sense how separate is the agent from its environment and why do we apply this distinction re self-improvement?
animals and humans do the same thing and are great at it, without 'self-improvement' with emphasis on the 'self'