VB6 Class Method Pointers


Author: Dave
Date: 07.27.15 - 9:08am



So I am working on a project that makes extensive use of callbacks from C code back to VB6 to pass data. Its an ActiveX control, which means that there can be multiple classes making use of these individual call backs.

Since all the calls are synchronous, and both vb/c code is single threaded, I am using a simple method to route the calls, transfer data, and raise the events back in the parent class. A very simplified example would look something like:

'module source
Dim activeClass As Class1

Public Sub callBack(x As Long)
    
    If activeClass Is Nothing Then Exit Sub
    
    activeClass.data = x
    activeClass.RaiseDataAvail
    
    
End Sub

'class1 source
Public data As Long

friend sub RaiseDataAvail()
    raiseevent DataAvailable()
End Sub


So this works and its fine. Really its the only way to do it (which is kind of what makes vb beautiful, you cant shoot yourself in the foot trying to get fancy) but! we can look around :)

So vb6 has a limitation that all call backs must be implemented in modules. Modules are not instanced and the code location is static for the life of the process. This makes it safe to give out the methods address using the addressof operator. They specifically denied access to the raw addresses of class functions because the class can go out of scope and then you have a dangling pointer and an extremely easy way to crash an application. Makes sense. Also, vb6 classes are actually COM objects and not straight forward C functions.

Still though, logically it seems like it would be nice IF we could use a function in a class as a call back method so the data transfer was per instance automatically. So can this be done if we accept we must be very responsible OR ELSE?


Edit: Sooo..below is where I went off the rails. I couldnt escape the addressof C style callback box i was thinking in, then I got all mired in details. The short story is...yes any of the vb6 classes can receive callbacks directly to them. You dont even want to think about addressof here. You just use the classes COM methods through IDispatch from your c++ code. Anyway I will leave the rest in place below because its still technically interesting.


Soo I was curious and looked into this. The answer is both yes and no. Technically, yes we can do this, BUT only when running in a compiled executable. In the IDE it appears to be different while once compiled it uses standard C++ classes as COM objects. (Maybe its possible in the IDE too, I dont know yet, I mean module callbacks are valid in IDE, more research is required, but if its already to much work to use if its not universal)

Anyway since I did the work, I will share the results because its interesting, but I do not consider this a useful mechanism because you can not utilize it in any sense while in the IDE and during code development and as of right now I do not have a reasonable fallback.

So just keep in mind the following is just trivia on VB6 internals.

Consider the following vb6 class:
Private Function dummy(x As Long)
    dummy = x
End Function

Public Function Method1(ByVal msg As Long, ByVal wParam As Long) As Long
    dummy &H11223344
    MsgBox "In method1! arg1:" & Hex(msg) & " arg2: " & Hex(wParam)
    Method1 = &HDDCCBBAA
End Function


Our mission is to find the raw address of Method1 at runtime, call it, and receive its return value through the equivalent of a callback.

The call to dummy with a magic hex number, is so that we can do a memory scan and find the target function.

So lets start with our main code with:

Dim c As New Class1

Private Sub Form_Load()
    msgbox objptr(c)
End Sub


(We use a form level class, so it doesnt go out of scope)

Next we look at this in the debugger:

**** Note this memory analysis is only valid in compiled code, does not work in IDE.. ****

objptr(c) = 14E880

0014E880  00403360  OFFSET Project1.__vba


00403360 >73426C5C MSVBVM60.BASIC_CLASS_QueryInterface 00403364 73426C28 MSVBVM60.Zombie_AddRef 00403368 7342EA13 MSVBVM60.Zombie_Release 0040336C 73485358 MSVBVM60.BASIC_DISPINTERFACE_GetTICount 00403370 734E962C MSVBVM60.BASIC_DISPINTERFACE_GetTypeInfo 00403374 734E9F0E MSVBVM60.BASIC_CLASS_GetIDsOfNames 00403378 734E9FAA MSVBVM60.BASIC_CLASS_Invoke 0040337C 004014D5 Project1.004014D5 00403380 004014E2 Project1.004014E2 00403384 004014C8 Project1.004014C8

004014C8 E9 530B0000 JMP 00402020 ; Class1::dummy (private) .... 004014D5 E9 E60B0000 JMP 004020C0 ; Class1::Method1 (public) .... 004014E2 E9 790C0000 JMP 00402160 ; Class1::Method2 (public)



Class1::Method1 ------------------- 004020C0 > 55 PUSH EBP 004020C1 8BEC MOV EBP,ESP 004020C3 83EC 0C SUB ESP,0C 004020C6 68 16114000 PUSH 401116 ; JMP to MSVBVM60.__vbaExceptHandler 004020CB 64:A1 00000000 MOV EAX,DWORD PTR FS:[0] 004020D1 50 PUSH EAX 004020D2 64:8925 00000000 MOV DWORD PTR FS:[0],ESP 004020D9 83EC 20 SUB ESP,20 004020DC 53 PUSH EBX 004020DD 56 PUSH ESI 004020DE 57 PUSH EDI 004020DF 8965 F4 MOV DWORD PTR SS:[EBP-C],ESP 004020E2 C745 F8 F0104000 MOV DWORD PTR SS:[EBP-8],4010F0 004020E9 33FF XOR EDI,EDI 004020EB 897D FC MOV DWORD PTR SS:[EBP-4],EDI 004020EE 8B75 08 MOV ESI,DWORD PTR SS:[EBP+8] ;objptr(class instance) 004020F1 56 PUSH ESI 004020F2 8B06 MOV EAX,DWORD PTR DS:[ESI] ;vtable 004020F4 FF50 04 CALL DWORD PTR DS:[EAX+4] ;MSVBVM60.Zombie_AddRef 004020F7 8B0E MOV ECX,DWORD PTR DS:[ESI] 004020F9 8D55 D8 LEA EDX,DWORD PTR SS:[EBP-28] 004020FC 8D45 D4 LEA EAX,DWORD PTR SS:[EBP-2C] 004020FF 52 PUSH EDX 00402100 > 50 PUSH EAX 00402101 897D D4 MOV DWORD PTR SS:[EBP-2C],EDI 00402104 56 PUSH ESI 00402105 897D E8 MOV DWORD PTR SS:[EBP-18],EDI 00402108 897D D8 MOV DWORD PTR SS:[EBP-28],EDI 0040210B C745 D4 44332211 MOV DWORD PTR SS:[EBP-2C],11223344 <-- our marker


So Objptr holds the pointer to the COM objects VTable (function list). The First 7 entries are COM methods automatically added for us. Followed by a list of our user methods. Our user methods have been reordered from the source, public first then private. And these addresses are relative jump instructions to the start of the actual code. Looking at the code, we can scan its bytes to find our marker and identify the target method from the others.

So lets write a small tool to analyze memory and extract these values for us and find our target method.



We will attempt to call it using
Private Declare Function CallWindowProc Lib "user32" Alias "CallWindowProcA" ( _
    ByVal lpPrevWndFunc As Long, _
    ByVal hwnd As Long, _
    ByVal msg As Long, _
    ByVal wParam As Long, _
    ByVal lParam As Long _
) As Long


Now from my method1 prototype, you will notice that it only has 2 long arguments and returns a long. While CallwindowProc takes 5 arguments?! WTF right? Well here is where some trickery comes into play. The VB6 class functions expect a c++ this pointer to be passed in as the first argument. So thats is going to eat our first argument from callwindowproc. In IDA they look like this:

.text:004043E0 ; void __stdcall Class1::Method1(Class1 *this)


Also to handle a return value, the vb6 class method expects a pointer to the return value to be as the last argument. So this eats up our last arg. A layout of the arguments is below:
'call window proc arguments mapping:
'    ByVal lpPrevWndFunc As Long,  'raw address of our method
'    ByVal hwnd As Long,           'eaten by C++ this pointer
'    ByVal msg As Long,            'usable arg1
'    ByVal wParam As Long,         'usable arg2
'    ByVal lParam As Long          'eaten by address for return value


So our call to CallWIndowProc would have to look like this:
Private Sub Command3_Click()
    Dim this As Long
    Dim retVal As Long
    this = ObjPtr(c)
    CallWindowProc method1_raw_address, this, &HAAAAAAAA, &HBBBBBBBB, VarPtr(retVal)
    MsgBox "Method 1 returned: " & Hex(retVal)
End Sub


Now this does indeed work, but as mentioned before has no practical development use since you cant use it at all while developing your code.

Anyway it was an interesting experiment and makes an additional argument on why they dont allow you to use addressof on class methods.

And just out of curosity..here is how callbacks are handled while in the IDE. (note this thunk will only be hit if a client accesses the method through the address given out by addressof. Directly calling the method in VB does not route through here which adds to the thought that when in the IDE everything is PCode based)

00AB0D24   A1 200DAB00      MOV EAX,DWORD PTR DS:[AB0D20]
00AB0D29   0BC0             OR EAX,EAX
00AB0D2B   74 13            JE SHORT 00AB0D40
00AB0D2D   B8 AF16A90F      MOV EAX,0FA916AF
00AB0D32   FFD0             CALL EAX           ; VBA6.EbMode
00AB0D34   83F8 02          CMP EAX,2
00AB0D37   74 07            JE SHORT 00AB0D40
00AB0D39   B8 8E8E1E00      MOV EAX,1E8E8E
00AB0D3E   FFE0             JMP EAX
00AB0D40   33C0             XOR EAX,EAX
00AB0D42   C2 0400          RETN 4

001E6F1E   BA 18FD1E00      MOV EDX,1EFD18
001E6F23   B9 0B00C00F      MOV ECX,0FC0000B
001E6F28  -FFE1             JMP ECX           ; VBA6.ProcCallEngine





Comments: (2)

On 04.09.16 - 8:21am Dave wrote:
in hindsight, me looking at how addressof was implemented doesnt really matter, if the callback address is grabbed from the COM interface vtable, which is already built to interact with native code regardless of pcode, native, or ide..brain fart, my initial mistake was going after the raw implementation address of the function. So this research is back on..There is also an easy way to route events to a specific class from a single callback in a module.

On 09.27.22 - 8:25pm dave wrote:
this works nicely and has some nice tricks.. seems like there might be an easier way to go about it though..

 
Leave Comment:
Name:
Email: (not shown)
Message: (Required)
Math Question: 59 + 13 = ? followed by the letter: V 



About Me
More Blogs
Main Site
Posts: (All)
2024 ( 2 )
2023 ( 9 )
2022 ( 4 )
2021 ( 2 )
2020 ( 4 )
2019 ( 5 )
2018 ( 6 )
2017 ( 6 )
2016 ( 22 )
2015 (15)
     C# self register ocx
     VB6 Class Method Pointers
     Duktape Debug Protocol
     QtScript 4 VB
     Vb6 Named Args
     vb6 Addin Part 2
     VB6 Addin vrs Toolbars
     OpenFile Dialog MultiSelect
     Duktape Example
     DukTape JS
     VB6 Unsigned
     .Net version
     TitleBar Height
     .NET again
     VB6 Self Register OCXs
2014 ( 25 )
2013 ( 4 )
2012 ( 10 )
2011 ( 7 )
2010 ( 11 )
2009 ( 3 )