64bit IDA Plugins

Author: David Zimmer
Date: 10.16.17 - 7:48am

So as we know IDA is going to primarily be a x64 binary now.

Bit of a hiccup for me since I use VB6 for my gui's. With IDAJscript the IDE/script engine actually run out of process and use interprocess communication (IPC) to talk to a small plugin stub, IDASRVR, that runs within IDA.

IDASRVR is going to be my escape path I suppose. I will have to convert IDACompare and others to run fully out of process which is fine. More stable for me to have my code insulated from API changes in IDA anyway.

So I started on the conversion of IDASRVR to first bring in 64bit p64 support which it didnt have yet (while keeping it a 32bit compile). This had some interesting bits.

My quickcall mechanism went almost completely out the window for x64. Using only sendmessage I was able to pass in two 32bit arguments and receive a 32bit return value. Well thats out for anything that takes a 64bit address. (If I was an x64 calling process it would probably still be ok but I am not)

Ultimately I want to get my IDAJScript running with the new IDA. So I have 3 layers to get working. First my IDASRVR plugin stub running in IDA, second my VB6 layer, and third its integration with the Duktape javascript engine.

So apparently Javascript does not support precise 64 bit numbers. You need to use a Bigint library for that. I didnt see that coming. Duktape was a perfect choice for a script engine with its nice tight size (1mb) and its debugger protocol. It took me like 3 mos of hobby time to get that debugger working the way I wanted and another 3 mos to get the Scintilla IDE portion where I wanted it. It was a big project to get just those components working.

VB6 doesnt support 64 bit numbers natively either so here is the plan as best I see it right now.

I will port IDASRVR to work with the 64bit IDA. It will support plw and p64 and run in process. All my other plugins, at least those worth porting (At least IDACOmpare and IDAJScript) will run fully out of process and only deal with IDA through the IDASRVR layer. Developing plugins in process is no fun anyway, out of process plugin interface is way better on so many levels. For lack of a better term I guess I will call these plugouts.

There will be slight differences between the 32bit and 64bit api use. Since neither vb6 or js support 64 bit numbers I will have to use a BigInt class to deal with them.

I could just convert fully and use only 64 bit addresses in the vb6 and js client classes...but using the BigInt class is going to add some annoying extra steps for quick scripts, and honestly 95% of the time I am working on 32bit disassemblies still.

So the plan is I am going to keep my 32bit IDAClient class unchanged (good already debugged and stable and wont break existing code). For working with 64bit disassemblies, I will add a IDAClient64 class whos exposed interface is almost identical, except that any arguments or return values which work with addresses now use a ULong64 class type. This vb6 class for BinInt support will be usable on the JS layer as well killing two birds with one stone.

Down side, 64bit and 32bit scripts will not be compatible, although you can write one script that works with both. This is annoying but I am dealing with multiple layers of incompatibilities so it kind of is what it is. Plus I dont do that much work with 64bit disassemblies anyway so.

64bit transition in general has been annoying for me. This also includes having to deal with Windows file system/registry redirection and COM registration concerns.

Aghh yes complexities abound and multiply. The poor human mind drowning on its current technological path. A couple relevant examples from todays news here, and here. It appears that today us is not smart enough for tomorrow us, or maybe tomorrow us just doesnt like today us that much :)

Comments: (2)

On 10.16.17 - 9:32am Dave wrote:
Some people might ask why i bother..I could just save myself a ton of work and use the built in python like everybody else.

I really want the syntax highlighting, intellisense and code debugging features of IDAJscript though.

I have read you can get WingWare IDE ($100 personal/ $250 professional license) for python to work as the remote debugger for IDA python, but never tried it myself yet.

For context, when I first started this project IDA only had IDC scripting support with what you typed into a flat textbox.

Times have changed, but I am already pretty invested and like what I have built. I also really like a fully open source stack.

IDASRVR opens the door for lots of other tool integration such as:

scdbg being able to -sync shellcode emulation
IDACompare able to pull in decompilation results on the fly or jump to functions
custom wingraph replacements able to navigate and rename functions
ability to work with IDA through arbitrary compiled languages
ability to debug/develop plugouts in your debugger of choice without having to continually open/close IDA
At a min at least IDASRVR must be updated and having a remote control interface for IDA integration has had proven utility.

On 10.22.17 - 1:49am Dave wrote:
If you want to see a diff for what had to change to get IDASRVR2 to compile for IDA7

Leave Comment:

Name:

Email: (not shown)

Message: (Required)

Math Question: 98 + 41 = ? followed by the letter: W