VB6 Class Method Pointers
Date: 07.27.15 - 9:08am
So I am working on a project that makes extensive use of callbacks from C code back to VB6 to pass data. Its an ActiveX control, which means that there can be multiple classes making use of these individual call backs.
Since all the calls are synchronous, and both vb/c code is single threaded, I am using a simple method to route the calls, transfer data, and raise the events back in the parent class. A very simplified example would look something like:
So this works and its fine. Really its the only way to do it (which is kind of what makes vb beautiful, you cant shoot yourself in the foot trying to get fancy) but! we can look around :)
So vb6 has a limitation that all call backs must be implemented in modules. Modules are not instanced and the code location is static for the life of the process. This makes it safe to give out the methods address using the addressof operator. They specifically denied access to the raw addresses of class functions because the class can go out of scope and then you have a dangling pointer and an extremely easy way to crash an application. Makes sense. Also, vb6 classes are actually COM objects and not straight forward C functions.
Still though, logically it seems like it would be nice IF we could use a function in a class as a call back method so the data transfer was per instance automatically. So can this be done if we accept we must be very responsible OR ELSE?
Edit: Sooo..below is where I went off the rails. I couldnt escape the addressof C style callback box i was thinking in, then I got all mired in details. The short story is...yes any of the vb6 classes can receive callbacks directly to them. You dont even want to think about addressof here. You just use the classes COM methods through IDispatch from your c++ code. Anyway I will leave the rest in place below because its still technically interesting.
Soo I was curious and looked into this. The answer is both yes and no. Technically, yes we can do this, BUT only when running in a compiled executable. In the IDE it appears to be different while once compiled it uses standard C++ classes as COM objects. (Maybe its possible in the IDE too, I dont know yet, I mean module callbacks are valid in IDE, more research is required, but if its already to much work to use if its not universal)
Anyway since I did the work, I will share the results because its interesting, but I do not consider this a useful mechanism because you can not utilize it in any sense while in the IDE and during code development and as of right now I do not have a reasonable fallback.
So just keep in mind the following is just trivia on VB6 internals.
Consider the following vb6 class:
Our mission is to find the raw address of Method1 at runtime, call it, and receive its return value through the equivalent of a callback.
The call to dummy with a magic hex number, is so that we can do a memory scan and find the target function.
So lets start with our main code with:
(We use a form level class, so it doesnt go out of scope)
Next we look at this in the debugger:
**** Note this memory analysis is only valid in compiled code, does not work in IDE.. **** objptr(c) = 14E880 0014E880 00403360 OFFSET Project1.__vba
So Objptr holds the pointer to the COM objects VTable (function list). The First 7 entries are COM methods automatically added for us. Followed by a list of our user methods. Our user methods have been reordered from the source, public first then private. And these addresses are relative jump instructions to the start of the actual code. Looking at the code, we can scan its bytes to find our marker and identify the target method from the others.
So lets write a small tool to analyze memory and extract these values for us and find our target method.
We will attempt to call it using
Now from my method1 prototype, you will notice that it only has 2 long arguments and returns a long. While CallwindowProc takes 5 arguments?! WTF right? Well here is where some trickery comes into play. The VB6 class functions expect a c++ this pointer to be passed in as the first argument. So thats is going to eat our first argument from callwindowproc. In IDA they look like this:
.text:004043E0 ; void __stdcall Class1::Method1(Class1 *this)
Also to handle a return value, the vb6 class method expects a pointer to the return value to be as the last argument. So this eats up our last arg. A layout of the arguments is below:
So our call to CallWIndowProc would have to look like this:
Now this does indeed work, but as mentioned before has no practical development use since you cant use it at all while developing your code.
Anyway it was an interesting experiment and makes an additional argument on why they dont allow you to use addressof on class methods.
And just out of curosity..here is how callbacks are handled while in the IDE. (note this thunk will only be hit if a client accesses the method through the address given out by addressof. Directly calling the method in VB does not route through here which adds to the thought that when in the IDE everything is PCode based)
00AB0D24 A1 200DAB00 MOV EAX,DWORD PTR DS:[AB0D20] 00AB0D29 0BC0 OR EAX,EAX 00AB0D2B 74 13 JE SHORT 00AB0D40 00AB0D2D B8 AF16A90F MOV EAX,0FA916AF 00AB0D32 FFD0 CALL EAX ; VBA6.EbMode 00AB0D34 83F8 02 CMP EAX,2 00AB0D37 74 07 JE SHORT 00AB0D40 00AB0D39 B8 8E8E1E00 MOV EAX,1E8E8E 00AB0D3E FFE0 JMP EAX 00AB0D40 33C0 XOR EAX,EAX 00AB0D42 C2 0400 RETN 4 001E6F1E BA 18FD1E00 MOV EDX,1EFD18 001E6F23 B9 0B00C00F MOV ECX,0FC0000B 001E6F28 -FFE1 JMP ECX ; VBA6.ProcCallEngine
Comments: (1)On 04.09.16 - 8:21am Dave wrote: