Quick post on AI coding

I am going to do a quick informal post on AI coding.

So to give it a shot, I wanted to try a rather involved project. My personal needs had me wanting a custom-built JavaScript engine from scratch, and a new Scintilla wrapper OCX control. Both would be done in VB6.

After playing with several AIs, I settled on Claude.AI as my favorite.

Both projects were quite involved and came out very well:

https://github.com/dzzie/js4vb
https://github.com/dzzie/sci4vb

I will mostly talk about the JS engine since it’s the bigger project and more complex.

To start, I have been a VB6 developer for 20+ years. It’s still my main language.

I know the theory behind the stages of developing a JS engine, but I do not have the experience to write one from scratch and am not a master of the JS ES5 specification. I am leaning on Claude’s knowledge hard here.

If you're curious how we got here and why I bothered to build a custom engine from scratch... it’s because I’ve been hitting a wall for over two decades. This project is the culmination of years of trying to "bolt on" scripting to VB6 with varying degrees of success.

After 25 years, I knew exactly what I wanted:

• Native VB6: Built specifically for embedding.
• Seamless COM: Native IDispatch support.
• Integrated Debugger: No more hacky C-wrappers.
• Full 64 bit Number Support: No more 53 bit insanity

Problem, I just didn't have the 1,000+ hours required to hand-code it. Now with new tech, AI finally gave me the power to execute on a vision I've had since the late 90s.

First, I started by creating a JS AST parser. This is the foundation for a script interpreter and is a whole project all on its own. We built this up iteratively. I used jquery.js as my test suite and we kept working until it could be parsed successfully. We designed hundreds of tests and added tons of debugging output as we went. We finally got it stable. I then asked about corner cases and other common gotchas and we worked through these.

Turns out this is a perfect target for AI:

• clear JS specifications
• tons of known test cases
• dozens of stable implementations
• lots of literature written on the task.
• theres no way to hide junk code: it will blow up spectacularly.

Once the AST parser was solid, we then got to work on the interpreter and we started incrementally adding features.

Claude knew what order to add features and wrote extensive test case at each stage. I grilled it and added design decisions along the way. (The Git repo unfortunately does not represent this building process.)

We then started adding some fancy features like true 64-bit numbers (in a host language that doesn’t support them!) and seamless COM support. We even had to write some C DLLs for things that I would not attempt in VB6 itself.

We even went a little crazy adding an IDispatch proxy mechanism so VB6 could access native JS objects through the VB6 Object type.

Check out the project readme. The features are actually pretty impressive! This was about 50 hrs of coding, which is simply not a possible timeline in any other way regardless of team size, budget, or expertise. This level of productivity is only possible through AI. This should have taken a team of 4 a year solid.

So what did I learn?

• #1 holy shit – super impressed!
• Work in testable chunks and test the hell out of each element
• Use copious amounts of debug output so you can paste it back into the AI for analysis
• Psych it up, you must tell it you are building pro-grade software or it will be lazy
• You are the architect. If you don’t like what you’re seeing, correct it
• I preloaded it with the code components I knew we needed like a VB6-compatible 64 bit math lib
• You will find model weaknesses that will create repeatable classes of bugs you must warn it about
  • Do not Dim As New in a loop expecting new objects
  • It does not handle the Declare mechanisms well; you must train it per context
  • Do not let it try to use regex – at all
• It will magnify what you’re capable of, but it will produce a mountain of shit if not guided
• It can very easily duplicate code. It won’t remember what it did before in large contexts
• There are some types of bugs it simply can’t solve. You may need to intervene and probe deeply
• Some AI context instances will prefer to just show you edits to make . This can get very dangerous trying to weave in edits without enough surrounding context. ask for more!
• It will likely atrophy brains and make people lazy
• Code integrity will absolutely suffer trying to test code we don’t fully understand.
• Younger coders will never really learn to code as well as the older ones who had to suffer to learn
• complex problems may need to be run by other AI for consensus (fresh eyes applies!)

Ok, so now on to some practical concerns. To code this JS engine I easily burned through 18 different contexts. Each context has a length limit. As it gets “tired” or “full” it will start tweaking out. You need to recognize this and switch at a logical stop/start point.

When starting a new context you need to:

• reload all of the information it needs upfront
• instruct it to ask for more info as needed
• reinterate overall project goals and demand professionalism

So how do you reload a context without overwhelming it?

I have a code helper add-in for the VB6 IDE that I added some features to:

• Right-click replace function (with Unix2DOS built in)
• Right-click replace code module (with Unix2DOS)
• Go to line (VB IDE does not have this)
• Copy all public properties/methods/events

These features are a minimum for efficient work. The add-in already had enhanced search-all and code navigation capabilities.

When I start a new context, first thing I would do is dump all of the classes and their public prototypes. Then I would upload the relevant files and tell it exactly what we are working on.

As the files got bigger, it noticed it would only study the parts I asked about. It actually would not try to understand elements outside of that. It appears to try to protect its own context limits.

You have to keep a running map of the entire project in your head and a fair understanding of its architecture and what each class does. You can’t trust it to make all the decisions for you. It does a really good job, and was able to weave in some amazing features in very little code, but you still have to know what’s going on and make the proper suggestions and rejections.

It was a painful process and I had to be at the top of my game that entire time to pull it off, but it really came out well. I am actually amazed by it.

There were a couple parts I knew were possible, but were above my direct ability, such as the CallByNameEx and IDispatch proxy. Claude really nailed them. Accessing JS objects through IDispatch was particularly hairy to implement!

The danger with this is I was almost dealing in black magic. When operating outside your core capabilities, you lose the power of foresight and are (temporarily) at the mercy of the AI capabilities.

We ran into a bug deep into the heart of this code that AI could not solve. It had to do with updating a JS value through the COM proxy while still having it live in the JS engine. Finally I had the insight to not create a new CValue object, but instead update the existing one the JS engine already had a reference to.

Another danger: Imagine your kitchen table filled with a pile of paperwork covering every inch, several feet high. AI can instantly generate massive blobs of text where you have no idea what’s inside there. It’s overwhelming. It made lots of decisions you will never know about. It’s essentially a black box. You may grow familiar with it as you make additions. In the beginning it’s just a blob, but its got electrolytes. Depending on your use case this can be very dangerous.

Danger #3: there were things that it totally skipped or used placeholder stubs for that I only found much later. For example it decided to not implement any of the compound assignment or increment operators other than += and ++.

I only found this out once I started actually using the engine and had a for loop trigger an endless loop as the implementation did nothing. You will have no warning things like this don’t exist! You can only find them through testing.

Overall it was a very rewarding process. I now have the JS engine I have always wanted. This was a 20-year pain point for me. Finally relief!

Some other posts that may be of interest:

• Access JS objects with a COM proxy (dynproxy)
• Ability to extend and override COM object methods with JS
• Returning JS objects directly to the script engine from COM calls
• IDA JScript - the impetus for this whole endeavor...

Comments: (3)