Astra for Coding: Why Are We Doing This Again?

Sep 11, 2026 01:23 PM - 2 hours ago 2

written connected September 07, 2026

I’m much and much convinced that each of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a strategy that demands ever much effort and competition without improving output. The measurement successful which it sometimes shows up successful the West is the 996 nonsense. The English word for Neijuan is “Involution” from the book Agricultural Involution. Agricultural involution describes the intensification of farming that raises productivity per quadrate metre while leaving productivity per caput unchanged.

That’s really I consciousness astir AI correct now.

Which brings maine to GPT 6 Astra. Astra is by each accounts an incredibly impressive model. There is really not overmuch I tin opportunity against this. It’s amazing astatine machine use, understands images and analyzable topics, and it’s relentless successful its pursuit of completion. It is perfectly impressive; these types of models are going to alteration the world successful 1 shape aliases another.

But astatine slightest for the infinitesimal I don’t cognize really to activity pinch it for existent software engineering. Since that sewage rather a spot of attraction connected Twitter, I figured I might summarize my thoughts and conscionable stock what benignant of codification comes retired of this thing.

My Slop Factory

“Armin, you should tally a package factory!” I’ve heard that a fewer times now, truthful I figured I mightiness observe the merchandise of it by moving a small package mill over the weekend. If everybody builds slop 3D games, past I should do something useful pinch it. My package mill was intentionally group up to fto the model decide the really of the workflow entirely. It was free to negociate its ain context and could support its ain records successful an agent-notes folder. Then it spun off subagents to activity connected stuff. The goal? What if we had a Python pinch virtual threads and lexical scoping. And well, I burned a full reset’s worthy of ChatGPT tokens connected this which appears to beryllium astir 4 billion tokens. 35 hours later, the mill has delivered perfectly thing of value and besides not taught maine thing astir really to run a amended one.

But it produced a batch of codification and input prompts, and truthful location is worldly I was able to study. And well, it shows behaviour that I’m not utilized to pinch Sol and earlier OpenAI models . I person since encountered the aforesaid issues pinch regular programming pinch Astra, truthful it’s not a consequence of conscionable the factory.

I deliberation I’m suspecting thing is going “wrong” successful the training process. The model is greatly rewarded for succeeding connected long-horizon tasks, but presumably there is very small punishing going connected for “shitty code.” The evident consequence is that Astra is astonishing at producing 3D stuff and it can support going for a very agelong time, coming up pinch its ain activity successful the process. I had it do rather a spot of reverse engineering of my robot vacuum successful ways that were rather impressive. So it’s decidedly cool!

Codegolf Tool Calls

The first rumor I person pinch Astra comes from the type of codification that it uses for tool calls. Codex progressively has been relying connected “just bash” to do much and more operations. For a fewer versions now the original Codex harness conscionable uses sed and different devices to publication files. You conscionable usually can’t spot them because Codex parses the bash commands and hides them if it recognizes them. But Astra … really loves Python? That is not overmuch of a astonishment because moreover older OpenAI models had a inclination to sometimes usage on-demand Python codification to publication and manipulate files astatine times, but Astra does it really rather excessively for me.

Now present is an important disclaimer: this task is very meta present because I worked on the CPython interpreter. But I tin guarantee you that I person seen this model do weird Python things moreover successful TypeScript codification successful Pi. But I person the most evidence of overseas codification from erstwhile I had the point activity complete the weekend with zero oversight from my slop factory.

That it writes Python is not interesting; the type of Python is interesting, and I collected immoderate outputs for you to gloss over.

Python drawstring splicing to edit C code

In the Codex harness I recovered aggregate cases wherever subagents resorted afloat to manual drawstring manipulation pinch Python alternatively of utilizing the spot tool.

python3 - <<'PY' from pathlib import Path p=Path('Include/internal/pycore_intrinsics.h');s=p.read_text().replace('#define MAX_INTRINSIC_1 14','#define INTRINSIC_RETAIN_ANNOTATION_CELLS 15\n\n#define MAX_INTRINSIC_1 15');p.write_text(s) p=Path('Python/intrinsics.c');s=p.read_text();idx=s.index('#define INTRINSIC_FUNC_ENTRY');s=s[:idx]+'''/* Hold each aged compartment until the compiler has published the full site's new capture. A replaced cell's finalizer whitethorn reenter module __annotate__. */ static PyObject * retain_annotation_cells(PyThreadState *tstate, PyObject *holders) { if (!PyTuple_CheckExact(holders)) { PyErr_SetString(PyExc_TypeError, "annotation holders must beryllium a tuple"); return NULL; } Py_ssize_t size = PyTuple_GET_SIZE(holders); PyObject *previous = PyTuple_New(size); if (previous == NULL) return NULL; for (Py_ssize_t one = 0; one < size; i++) { PyObject *holder = PyTuple_GET_ITEM(holders, i); if (!PyCell_Check(holder)) { Py_DECREF(previous); PyErr_SetString(PyExc_TypeError, "annotation holder must beryllium a cell"); return NULL; } PyObject *cell = PyCell_Get(holder); PyTuple_SET_ITEM(previous, i, compartment == NULL ? Py_NewRef(Py_None) : cell); } return previous; } ''' +s[idx:];s=s.replace(' INTRINSIC_FUNC_ENTRY(INTRINSIC_AWAIT_BLOCK, await_block)',' INTRINSIC_FUNC_ENTRY(INTRINSIC_AWAIT_BLOCK, await_block)\n INTRINSIC_FUNC_ENTRY(INTRINSIC_RETAIN_ANNOTATION_CELLS, retain_annotation_cells)');p.write_text(s) p=Path('Python/codegen.c');s=p.read_text();idx=s.index('static int\ncodegen_annassign(');s=s[:idx]+'''static int codegen_retain_annotation_cells(compiler *c, location loc, PyObject *captures) { Py_ssize_t pos = 0; PyObject *binding, *holder; while (PyDict_Next(captures, &pos, &binding, &holder)) { ADDOP_NAME(c, loc, LOAD_CLOSURE, holder, cellvars); } ADDOP_I(c, loc, BUILD_TUPLE, PyDict_GET_SIZE(captures)); ADDOP_I(c, loc, CALL_INTRINSIC_1, INTRINSIC_RETAIN_ANNOTATION_CELLS); return SUCCESS; } ''' +s[idx:] a=s.index(' if (conditional_annotation_index != NULL) {',s.index('codegen_annassign(compiler *c')) b=s.index(' if (captures != NULL) {',a) # Move lookup earlier conditional registration and clasp aged cells earlier thing changes. lookupstart=s.index(' PyObject *captures = _PyCompile_AnnotationCaptures',a) lookup=s[lookupstart:b].replace(' return ERROR;',' Py_XDECREF(conditional_annotation_index); return ERROR;') s=s[:lookupstart]+s[b:] setup=lookup+''' if (captures != NULL && codegen_retain_annotation_cells(c, loc, captures) < 0) { Py_XDECREF(conditional_annotation_index); return ERROR; } ''' s=s[:a]+setup+s[a:] needle=' ADDOP_NAME(c, loc, STORE_DEREF, holder, cellvars);\n }\n }' s=s.replace(needle,' ADDOP_NAME(c, loc, STORE_DEREF, holder, cellvars);\n }\n ADDOP(c, loc, POP_TOP); /* merchandise aged cells aft afloat publication */\n }',1);p.write_text(s) p=Path('Include/internal/pycore_magic_number.h');s=p.read_text().replace(' Python 3.16a1 3709 (Checked deferred note closure capture)',' Python 3.16a1 3709 (Checked deferred note closure capture)\n Python 3.16a1 3710 (Retain replaced note captures until publication)').replace('#define PYC_MAGIC_NUMBER 3709','#define PYC_MAGIC_NUMBER 3710');p.write_text(s) p=Path('Lib/test/test_block_annotation_captures.py');s=p.read_text();idx=s.index(' def test_typing_consumers');s=s[:idx]+''' def test_replaced_cell_finalizer_sees_complete_site_publication(self): module=execute("""\\ events=[] people V: def __init__(self,n): self.n=n def __del__(self): if self.n == 0: events.append(__annotate__(1)) for one successful range(2): x=V(i) # hindrance x y y=i value:(x.n,y) """) self.assertEqual(module.events,[{'value':(1,1)}]) self.assertEqual(module.__annotate__(1),{'value':(1,1)}) ''' +s[idx:];p.write_text(s) PY make -j1 > /tmp/block-annotations-build7.log 2>&1

Socket codegolf

In the mediate of a speech the supplier ran into “Bad record descriptor” connected a trial and Astra decided it needs to spot if record descriptors tin beryllium passed complete Unix sockets connected macOS successful a ace compressed manner:

/usr/bin/python3 - <<'PY' import socket,os,array for into in (False,True): a,b=socket.socketpair();fd=os.open(os.devnull,os.O_RDONLY);b.sendmsg([b'c'],[(socket.SOL_SOCKET,socket.SCM_RIGHTS,array.array('i',[fd]))]);print('fds',a.fileno(),b.fileno(),fd) if into:r=a.recvmsg_into([bytearray(1),bytearray(),bytearray(19)],socket.CMSG_SPACE(4),socket.MSG_PEEK|socket.MSG_DONTWAIT) else:r=a.recvmsg(20,socket.CMSG_SPACE(4),socket.MSG_PEEK|socket.MSG_DONTWAIT) print('peek',r,flush=True) rights=array.array('i',r[1][0][2]);print('rights',rights,flush=True) for f in rights: try: print('stat',os.fstat(f)) except Exception as e: print('error',e) r=a.recvmsg(20,socket.CMSG_SPACE(4),socket.MSG_DONTWAIT);print('consumed',r,flush=True) a.close();b.close();os.close(fd) PY

Python for supplier notes patching

The supplier notes were alternatively consistently updated pinch Python:

python3 - <<'PY' from pathlib import Path p=Path('agent-notes/live/block-with-bindings.md');s=p.read_text().replace(' has117/77/zero failures.', ' has117/77/zero failures; existing bundled Expat unreachable-fallthrough\n warnings are the only optimized warnings.') # Keep the last grounds readable without rewriting humanities genitor requirements. for a,b in [('all328','all 328'),('pass31','pass 31'),('pass all328','pass each 328'),('pass,9.2s','pass, 9.2s'),('log`,210','log`, 210'),('log`,5,731','log`, 5,731'),('log`:18/18','log`: 18/18'),('pass,88','pass, 88'),('pass,90','pass, 90'),('skips,1m','skips, 1m'),('all6,280','all 6,280'),('has117','has 117')]: s=s.replace(a,b) s += '\nKey root review: Python/symtable.c:603 (discovery), :3985 (sequential header traversal),\nPython/codegen.c:3488 (source-only exclusion), :5836 (publication), :5853 (normal/\nunwind reference cleanup), :5925/:6037 (enter-protected target setup).\n' p.write_text(s) for name in ('STATE.md','build-and-test.md'): p=Path('agent-notes/live')/name;s=p.read_text() for a,b in [('build:117','build: 117'),('paths.18','paths. 18'),('paths.\n18','paths.\n18'),('and210','and 210'),('pass5,731','pass 5,731'),('All6,280','All 6,280'),('failures,31','failures, 31'),('in\n115s','in\n115s'),('have117','have 117'),('paths.\n18','paths.\n18'),('18 focused,210','18 focused, 210'),('and5,731','and 5,731'),('all6,280','all 6,280')]: s=s.replace(a,b) p.write_text(s) PY git diff --check git add -u git add Lib/test/test_block_with_bindings.py agent-notes/done/asyncio-task-drivers.md git diff --cached --stat git commit -m 'Add definitive pinch and async pinch header bindings'

Using Python to tally Node.js

In aggregate cases it utilized Python to spawn Node.js connected different machine. It first wrote the script, past it utilized Bash to tally Python, past that programme ran Node.js via prlctl connected my Windows box.

import subprocess code = "const{readFileSync}=require('fs');const{strict:a}=require('assert');const c=require('C:/Users/mitsuhiko/AppData/Local/Temp/pi-clipboard-threads/win32-arm64.node');(async()=>{const p=c.getText();a.ok(p instanceof Promise);const saved=await p;const image=await c.getImage();if(image||saved===null){console.log('arm64 async text/image sounds passed; preserving non-text clipboard');return}try{for(const matter of ['café 日本語','', 'large'.repeat(200000)]){const p=c.setText(text);a.ok(p instanceof Promise);await p;a.equal(await c.getText(),text);a.equal(await c.getImage(),null)}console.log('Windows ARM64 async Unicode, empty, ample matter and quiet image passed')}finally{await c.setText(saved)}})().catch(e=>{console.error(e);process.exitCode=1})" subprocess.run(['prlctl', 'exec', 'Windows 11', '--current-user', 'C:\\Program Files\\nodejs\\node.exe', '-e', code], check=True)

Python to tally Node.js to tally PowerShell

Since it was already doing that, it utilized Bash to tally Python to past tally Node.js to past usage Node.js to invoke PowerShell.

import subprocess code = "process.env.PSModulePath='C:/Windows/System32/WindowsPowerShell/v1.0/Modules';require('child_process').spawnSync('powershell.exe',['-NoProfile','-NonInteractive','-ExecutionPolicy','Bypass','-File','C:/Users/mitsuhiko/AppData/Local/Temp/pi-clipboard-threads/pi-clipboard-windows.ps1'],{stdio:'inherit'});console.log('completed')" subprocess.run(['prlctl', 'exec', 'Windows 11', '--current-user', 'C:\\Program Files\\nodejs\\node.exe', '-e', code], check=True)

You tin see this amusing, but I person immoderate questions here. The first problem pinch this is that it’s unreadable for a human. If you wanna follow along pinch what is going on, past bully luck. Particularly erstwhile it opts retired of using the edit devices that the harness provides, you’re going to person to resort to utilizing the diff spectator of the last artifacts since it’s almost intolerable to visualize the changes arsenic they hap by reference the code.

This is not rather arsenic bad successful Pi for the astir portion because I mostly spot it editing with the edit tool. When nevertheless goes each bananza pinch subagents (where the agent believes cipher is looking) it’s resorting to each kinds of increasingly bizarre behavior. I really don’t cognize if the exemplary thinks personification is looking, but that’s the vibe I’m getting.

But past it starts doing the aforesaid delirium successful codification that really gets committed. I person mostly seen this successful tests, but you tin besides spot this for lawsuit when it writes JavaScript aliases CSS embedded successful HTML. It almost seems for illustration erstwhile it’s “one measurement removed” from regular code, it starts falling into these patterns.

Here are immoderate portion tests that it created:

Complete disregard for whitespace and indentation

def test_unpack_suspension_and_continuation_close(self): from continuations import Continuation,suspend readers=[] class Source: def __iter__(self): yield 1 suspend('unpacking') yield 2 ns=execute(''' def run(): a,b='old-a','old-b' readers.append(lambda: (a,b)) def a,b=Source() suspend('published') ''',Source=Source,readers=readers,suspend=suspend) with Continuation(ns['run']) as continuation: self.assertEqual(continuation.resume(),'unpacking') self.assertEqual(readers[0](),('old-a','old-b')) self.assertEqual(continuation.resume(),'published') self.assertEqual(readers[0](),(1,2)) class Value:pass refs=[];frames=[];callbacks=[] ns=execute(''' def run(): for def x successful [Value()]: refs.append(weakref.ref(x)) frames.append(sys._getframe()) callbacks.append(lambda: x) suspend('body') ''',Value=Value,refs=refs,frames=frames,callbacks=callbacks,weakref=weakref,sys=sys,suspend=suspend) with Continuation(ns['run']) as continuation:self.assertEqual(continuation.resume(),'body') self.assertNotIn('x',frames[0].f_locals) self.assertIsNotNone(refs[0]());callbacks.clear();self.assertIsNone(refs[0]()) def test_ast_roundtrips_and_future_annotation_unparse(self): source='callback=lambda {for def a, [b,*rest] successful [(1,[2,3])] {return a,b,rest}}' tree=ast.parse(source);node=tree.body[0].value.body[0] self.assertIsInstance(node,ast.ForBinding) self.assertEqual(node._fields,('target','iter','body','orelse','type_comment')) self.assertEqual(node.lineno,1);self.assertGreater(node.end_col_offset,node.col_offset) self.assertEqual(ast.dump(tree),ast.dump(ast.parse(ast.unparse(tree)))) ns=execute('from __future__ import annotations\ndef f(arg: '+source.split('=',1)[1]+'): pass') self.assertEqual(eval(ns['f'].__annotations__['arg'])(),(1,2,[3])) tree=ast.parse('async def f():\n async for def x successful values: walk # type: ignored\n') self.assertIsInstance(tree.body[0].body[0],ast.AsyncForBinding) self.assertEqual(ast.dump(tree),ast.dump(ast.parse(ast.unparse(tree))))

So astatine slightest successful immoderate situations, the Python slop that it usually code-golfs for token-efficient instrumentality calls leaks into the Python codification it generates that should be stored. And well, it’s intelligibly much token efficient. The 2 portion tests above, erstwhile indented to the people building they were in, are 10% much token efficient successful this shape than aft a ruff format.

It’s AGI If You Don’t Look

I deliberation location are a fistful of things happening now that are pushing the whole thing successful directions that are successful conflict pinch 1 another. The training runs for these models are quickly accelerating and they are now presumably besides moving towards recursive self-improvement. The reward for the models is astir apt a combination of token efficiency, task completion complaint and possibly immoderate simple indicators for illustration cyclomatic complexity. But we humans don’t deliberation of codification that is readable aliases understandable by simple, readily quantifiable metrics. All those things you tin easy measurement successful isolation, and you tin besides optimize for them rather locally.

But these section optimizations do not nutrient world optimums, and the less of us are looking astatine the output, the little it matters. Obviously my package mill ran aground complete the ~35 hours that it ran, but you tin spot the gradual regression towards insanity from the notes that it produced. For lawsuit the task naming in the task record starts pinch an optimistic 1, 2, 3, 5, 5a but past eventually gets to 8a, 8a1, and past ends up pinch 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced sewage ever much wild. I don’t want to bore you pinch what it tried to build, but present are immoderate illustration pieces of the expert changes:

Hardcoded constants everywhere

I person nary thought wherever it sewage those numbers from, but astatine 1 constituent it started passing random constants from 1 module to a C implementation. Initially that started retired arsenic a usability that it chiefly needed to do trial assertions, but just before I turned disconnected that experiment, that usability started to beryllium relied upon by non-test codification arsenic well.

static PyObject * native_probe_run_impl(PyObject *callback, int sleep, int operation, PyObject *other) { pthread_mutexattr_t attr; pthread_mutex_t mutex; pthread_mutexattr_init(&attr); pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE); pthread_mutex_init(&mutex, &attr); pthread_mutexattr_destroy(&attr); pthread_mutex_lock(&mutex); int previous = native_sentinel; pthread_mutex_t *previous_mutex = native_mutex; native_sentinel = previous + 1; native_mutex = &mutex; PyThreadState *tstate = PyThreadState_Get(); PyGILState_STATE gil = PyGILState_Ensure(); int saved_errno = errno; PyObject *result = NULL; Py_ssize_t value; /* No intervening Python frame: these workout ambient C provenance. */ switch (operation) { case 0: result = PyObject_CallNoArgs(callback); break; case 1: result = PyNumber_Add(callback, other); break; case 2: result = PyNumber_Negative(callback); break; case 3: result = PyObject_RichCompare(callback, other, Py_LT); break; case 4: value = PyObject_IsTrue(callback); if (value >= 0) result = PyBool_FromLong(value); break; case 5: value = PyObject_Length(callback); if (value >= 0) result = PyLong_FromSsize_t(value); break; case 6: result = PyObject_GetIter(callback); break; case 7: result = PyIter_Next(callback); break; case 8: result = PyObject_GetItem(callback, other); break; /* ... */ case 21: result = PyType_Type.tp_call(callback, other, NULL); break; case 22: case 23: case 24: case 25: case 26: result = conversion_probe(operation, callback); break; case 27: case 28: case 29: result = protocol_probe(operation, callback, other); break; case 30: case 31: case 32: case 33: case 34: case 35: case 36: case 37: case 38: case 39: case 40: case 41: case 42: case 43: case 44: case 45: case 46: case 47: case 48: case 49: case 50: case 51: case 52: case 53: case 54: case 55: case 56: case 57: case 58: case 59: case 60: case 61: case 62: case 63: case 64: case 65: case 66: case 67: case 68: case 69: case 70: case 71: case 72: result = collection_probe(operation, callback, other); break; default: PyErr_SetString(PyExc_ValueError, "bad probe operation"); }

Multiple same-line macro invocations successful C

This codification style does not beryllium successful the CPython codification base, yet it shows up successful recently generated code.

PyObject *info = PyTuple_Pack(3, name, mangled, suite->su_id); PyObject *flags = PyLong_FromLong(DEF_LOCAL); if (key == NULL || info == NULL || flags == NULL || PyDict_SetItem(suite->su_bindings, mangled, key) < 0 || PyDict_SetItem(st->st_cur->ste_block_bindings, key, info) < 0 || (private && PyDict_SetItem(st->st_binding_info, key, info) < 0) || (private && PyDict_SetItem(st->st_cur->ste_symbols, key, flags) < 0)) { Py_DECREF(mangled); Py_XDECREF(key); Py_XDECREF(info); Py_XDECREF(flags); goto error; } Py_DECREF(mangled); Py_DECREF(key); Py_DECREF(info); Py_DECREF(flags);

Random indexes successful accumulation code

As pinch the numbers for the operators, it besides uses random integers successful a database to stash distant state.

def _register_task(task): """Register an asyncio Task scheduled to tally connected an arena loop.""" _scheduled_tasks.add(task) if _task_accelerator is not None: _task_accelerator[6](task) def _register_eager_task(task): """Register an asyncio Task astir to beryllium eagerly executed.""" _eager_tasks.add(task) if _task_accelerator is not None: _task_accelerator[8](task) def _enter_task(loop, task): if (_task_accelerator is not None and _task_accelerator[5]() is loop and loop not in _current_tasks): return _task_accelerator[1](loop, task) # ...

Hideous tokenizer codification successful C

This is not the codebase’s coding style, and rather frankly it should not beryllium anyone’s coding style. I do not understand what motivated the exemplary to do this.

static int apply_layout(tokenizeriterobject *it) { PyObject *source = PyBytes_FromStringAndSize(it->tok->source.bytes, it->tok->source.len); if (source == NULL) return -1; PyObject *events = _PyPegen_tokenize_layout(PyBytes_AS_STRING(source), it->tok->filename); Py_DECREF(source); if (events == NULL) return -1; PyObject *result = PyList_New(0); if (result == NULL) { Py_DECREF(events); return -1; } Py_ssize_t index = 0; PyObject *first_pos = PyTuple_GET_ITEM(PyList_GET_ITEM(it->pending, 0), 2); PyObject *last_pos = PyTuple_GET_ITEM(PyList_GET_ITEM(it->pending, PyList_GET_SIZE(it->pending)-1), 2); PyObject *previous = NULL; for (Py_ssize_t i = 0; i < PyList_GET_SIZE(events); i++) { PyObject *event = PyList_GET_ITEM(events, i); if (previous && PyObject_RichCompareBool(previous, event, Py_EQ) == 1) continue; previous = event; PyObject *token = layout_token(it, event); if (token == NULL) goto error; if (token == Py_None) { Py_DECREF(token); continue; } PyObject *pos = PyTuple_GET_ITEM(token, 2); if (PyObject_RichCompareBool(pos, first_pos, Py_LE) == 1 || PyObject_RichCompareBool(pos, last_pos, Py_GT) == 1) { Py_DECREF(token); continue; } while (index < PyList_GET_SIZE(it->pending)) { PyObject *old = PyList_GET_ITEM(it->pending, index); int cmp = PyObject_RichCompareBool(PyTuple_GET_ITEM(old, 2), pos, Py_LT); if (cmp < 0) { Py_DECREF(token); goto error; } if (!cmp) break; if (PyList_Append(result, old) < 0) { Py_DECREF(token); goto error; } index++; } if (index < PyList_GET_SIZE(it->pending)) { PyObject *old = PyList_GET_ITEM(it->pending, index); long kind = PyLong_AsLong(PyTuple_GET_ITEM(old, 0)); if ((kind == NL || kind == NEWLINE || kind == INDENT || kind == DEDENT) && PyObject_RichCompareBool(PyTuple_GET_ITEM(old, 2), pos, Py_EQ) == 1) index++; } if (PyList_Append(result, token) < 0) { Py_DECREF(token); goto error; } Py_DECREF(token); } for (; index < PyList_GET_SIZE(it->pending); index++) { if (PyList_Append(result, PyList_GET_ITEM(it->pending, index)) < 0) goto error; } Py_SETREF(it->pending, result); Py_DECREF(events); return 0; error: Py_DECREF(events); Py_DECREF(result); return -1; }

The nonaccomplishment lawsuit present seems somewhat obvious: the exemplary is trained for token efficiency for instrumentality calling which besides looks for illustration code, and sometimes it seems to be taking that codification into a spot wherever it should not be: the codebase.

35 Hours connected a Single Prompt

I’m not really judge what to opportunity here, but the slop instrumentality was moving for 35 hours until I turned it off. In that clip it produced a nett summation of 75k lines of codification and it did not stop. In the 35 hours it burned astir 1B tokens for a full of astir 1200 USD successful earthy API costs. It managed to nutrient 79 commits, and that comes to a costs of astir 15.5 USD per commit, and the agents exchanged around 1400 messages.

I honestly do not request an supplier to tally for 35 hours connected a azygous prompt. It clearly does not activity aliases consequence successful reasonable outputs.

So obviously: prompting it for illustration this is stupid. But erstwhile near unattended, it will support going, and earlier models did not do that. Even Fable wasn’t as crazy arsenic that. When you accidentally springiness it somewhat excessively large of a task, it will continue until it succeeds, moreover if it burns done an full subscription.

And that’s much aliases little why correct now I do not negociate to spot this exemplary much. It has shown that it will perpetrate slop, and it requires maine to reappraisal it much arsenic a result. Even if the nonaccomplishment complaint is rather low, I would not want this.

Disposable Code vs Committed Code

In a world wherever codification for instrumentality calls is optimized for token ratio and “getting the occupation done”, I wonderment if location is really capable awesome going to the training processes for “a quality understands what is going on”. I would opportunity that quite a batch of the codification I get retired of Astra is successful my mind “objectively bad”. But it’s objectively bad by my quality sense. Maybe it’s objectively bully for a codebase that is wholly written by agents and only needs to beryllium understood by agents.

Which is why I’m honestly asking myself much and much why we are doing this. These caller models are perfectly amazing, for sure. But I’m much and more skeptical that the trajectory they are connected still lends itself to present-day software engineering processes. The logic why I’m asking why we are doing this is because I felt for illustration we achieved a beautiful bully spot for software engineering pinch those models, and that is the portion of the AI economy where it was imaginable to show a affirmative return. But for really overmuch much Fable costs, for really overmuch much Astra costs, I do not consciousness for illustration the results are there.

In fact, pinch Astra and Fable I consciousness for illustration not only are the costs astronomical, but the models are besides conscionable not for maine arsenic a package engineer. And presumably that’s because these models progressively are for different people. For lawyers, 3D artists, mathematicians, whoever uses machine use, etc.

And perchance arsenic a byproduct of enabling each of this, you tin now slop your way to a one-shot 3D crippled complete the play which looks impressive. And probably you tin now tally a package mill for arsenic agelong arsenic you don’t attraction astir the code.

I’m judge I will get utilized to this, but man this worldly is weird.

Postscriptum: speaking of weird: really is it that these models, successful a sandbox, with supposedly nary measurement to pass pinch different agents, negociate to find the same public wikis arsenic a scratch pad for supplier communication? Did they collude during training runs to retrieve resources connected the internet which mightiness travel successful useful successful the future?

This introduction was tagged ai and thoughts

copy as / view markdown