how pcode works Pt1

Author: David Zimmer
Date: 08.29.19 - 11:04am

This is just a short post to rehash the basics of how MS vb6 Pcode execution works within a pcode stream. The vb6 runtime has 6 contiguous tables of long pointers, each 255 entries long that hold a series of pointers to native opcode handlers. For the reference dll I have been using the addresses are as follows:

ENGINE:66106D14 _tblByteDisp          (table 0 - no lead byte)
ENGINE:66107114 _tblDispatch_Lead0    (table 1 - lead 0xFB)
ENGINE:66107514 _tblLead1             (table 2 - lead 0xFC)
ENGINE:66107914 tblLead2              (table 3 - lead 0xFD)
ENGINE:66107D14 tblLead3              (table 4 - lead 0xFE)
ENGINE:66108114 tblLead4              (table 5 - lead 0xFF)

Note that the version linked to above comes with debug symbols, including the opcode handler function names (which were not often included).

Now consider the following disassembled pcode:

401F93    89 3400               MemLdI2  [pr + 0x34]
401F96    F4 FF                 LitI2_Byte 255
401F98    FE5D 0200             OpenFile output

To see how execution flows, lets start at the end of MemLdI2 handler where it is getting ready to transfer execution to LitI2. In the opcode handlers, ESI always points to the current position within the pcode byte stream. Here our Esi start value will be 401F94.

ENGINE:66105FA5   xor     eax, eax       ;zero out eax
ENGINE:66105FA7   mov     al, [esi+2]    ;skip arg bytes and load next opcode index  (F4)
ENGINE:66105FAA   add     esi, 3         ;increment esi to next byte after
ENGINE:66105FAD   jmp     ds:_tblByteDisp[eax*4] ;jump to offset in pointer table

For the jump, the code from olly may be more clear:

66105FAD JMP DWORD PTR DS:[EAX*4+66106D14] ;Eax = F4 
DS:[661070E4]=66105CAB   MSVBVM60.lblEX_LitI2_Byte 
mem: 661070E4  66105CAB  MSVBVM60.lblEX_LitI2_Byte

Now when LitI2 goes to transfer execution to OpenFile, there is an extra step involved. Here it will go through the same exact process as above, but the opcode handler for FE will actually run a small stub that index into a whole new function table.

66105CB7 JMP DWORD PTR DS:[EAX*4+66106D14] ; Eax=FE -> MSVBVM60.lblBEX_Lead3

ENGINE:66106C29 _lblBEX_Lead3                                              
ENGINE:66106C29    xor     eax, eax
ENGINE:66106C2B    mov     al, [esi]    ;read current byte
ENGINE:66106C2D    inc     esi          ;increment esi to point to next byte
ENGINE:66106C2E    jmp     ds:tblLead3[eax*4]  ;jump to index eax in new table

So offset tblByteDisp[0xFE] leads execution to a small trampoline that reads the next byte, then uses that to jump to the final opcode handler for that index on tblLead3. If we look at the last few entries of the first table we see the following:

ENGINE:66107100     dd offset _lblBEX_Lead0
ENGINE:66107104     dd offset _lblBEX_Lead1
ENGINE:66107108     dd offset _lblBEX_Lead2
ENGINE:6610710C     dd offset _lblBEX_Lead3
ENGINE:66107110     dd offset _lblBEX_Lead4
ENGINE:66107114 _tblDispatch

This is the manner in which the pcode bytes are interpreted by the runtime and how they control execution flow of the native code from one pcode instruction to the next.

This is only a small slice of the story though. What happens when a pcode function wants to call another function? This is where the story deepens and things get interesting.

Calls to other functions work through the various native handlers. It determines where to send execution based on arguments embedded within the pcode bytestream itself as well as arguments pushed onto the stack by previous opcode operations (such as pushing a live object reference onto the stack). The byte stream arguments themselves additional can contain various types of hard coded data such as vtable offsets to call, expected stack adjustments from the call, and indexes into the constant pool to pull more complex data from.

There are numerous types of pcode call instructions such as the following:

0,&HD,VCallHresult  
2,&H69,VCallHresult 
0,&HE,VCallFPR8      
4,&H19,VCallFPR8
4,&H1A,VCallFPR8
4,&H1C,VCallFPR8
5,&H2B,VCallFPR8
0,&HF,VCallAd         
4,&H10,VCallUI1
4,&H11,VCallStr
4,&H12,VCallStr
4,&H17,VCallStr
4,&H18,VCallStr
4,&H13,VCallR4
4,&H14,VCallR8
4,&H15,VCallCy
4,&H1D,VCallCbFrame

0,&H10,ThisVCallHresul
2,&H6A,ThisVCallHresult
0,&H11,ThisVCallFPR8
0,&H12,ThisVCallAd
4,&H0,ThisVCallUI1
4,&H1,ThisVCallI2
4,&H2,ThisVCallI2
4,&H7,ThisVCallI2
4,&H8,ThisVCallI2
4,&H3,ThisVCallR4
4,&H4,ThisVCallR8
4,&H5,ThisVCallCy
4,&H9,ThisVCallHidden
4,&HA,ThisVCallHidden
4,&HC,ThisVCallHidden
5,&H2C,ThisVCallHidden
4,&HD,ThisVCallCbFrame

0,&H9,ImpAdCallHresult 
4,&H4F,ImpAdCallHresult
0,&HA,ImpAdCallFPR4 
4,&H29,ImpAdCallFPR4   
4,&H2A,ImpAdCallFPR4    
4,&H2C,ImpAdCallFPR4    
0,&HB,ImpAdCallI2 
0,&HC,ImpAdCallCy
4,&H25,ImpAdCallCy
0,&H5E,ImpAdCallI4  
4,&H21,ImpAdCallI4
4,&H22,ImpAdCallI4
4,&H27,ImpAdCallI4
4,&H28,ImpAdCallI4
4,&H23,ImpAdCallR4
4,&H24,ImpAdCallR8
4,&H20,ImpAdCallUI1
4,&H2D,ImpAdCallCbFrame
5,&H1E,ImpAdCallNonVirt

0,&H57,LateMemLdVar
4,&H99,LateMemLdVar
4,&H9B,LateMemSt  
4,&H9C,LateMemCallSt
4,&H9D,LateMemStAd
4,&H9A,LateMemCallLdVar
4,&H98,LateMemCall  
4,&HA6,LateMemNamedCall
4,&HA7,LateMemNamedCallLdVar
4,&HA8,LateMemNamedCallSt
4,&HA9,LateMemNamedStAd
4,&HA0,LateIdCall    
4,&HA4,LateIdCallSt
4,&HA2,LateIdCallLdVar   
0,&H2C,LateIdSt    
4,&HA3,LateIdSt
4,&HA5,LateIdStAd
0,&H61LateIdLdVar  
4,&HA1,LateIdLdVar
4,&HAA,LateIdNamedCall
4,&HAB,LateIdNamedCallLdVar
4,&HAC,LateIdNamedCallSt
4,&HAD,LateIdNamedStAd

5,&H44,VarLateMemCallSt
5,&H42,VarLateMemCallLdVar
5,&H3E,VarLateMemCallLdRfVar  
5,&H41,VarLateMemLdVar
5,&H3D,VarLateMemLdRfVar
5,&H43,VarLateMemSt  
5,&H45,VarLateMemStAd

Each of these take their own set of arguments that you have to understand and know how to decode. Examples of how to decode ImpAdCallNonVirt and LateIDCall are already available.

In future posts we will detail more nuances of how the system operates, how api are called, how call backs are handled, how pcode functions return to native code, how native code first starts executing pcode streams, and what the groupings above signify.

Comments: (1)

On 09.21.19 - 8:04am Dave wrote:
I dont really have a great place to put this, so I will just throw it in here as an interesting tidbit. Here is an excerpt from an old MSDN article. I mirror the interesting bit in case it ever disappears.

Any code that you ran in the IDE ran as Pcode and used VBA6.dll rather than msvbvm60.dll. This wasn’t the same as MSVBVM60 in quite a few ways but the important one here is that it had a lot more hooks for debugging. The Pcode engine generally did a good job of pretending to be the proper runtime system and contained a great deal of the same code. When you ran in the IDE, your app wasn’t really its own process but that hardly ever showed. When your component was not intrinsically runable in isolation (because it was a DLL or OCX), the VB6 IDE hosted it for you. That isn’t to say that VB6 was the client. Another application would be the client and would be pretty much unaware that VB6.exe was sitting in the middle. To do this, the IDE played fast and loose with the registry and fixed it up so that a component was instantiating an out of process server (VB6 itself) while the component being debugged thought that it had an inprocess client. In NT4, this allowed you to debug a component nominally hosted in MTS within the IDE although it sometimes required a few small changes.

If you are familiar with COM (another fine legacy technology), you will know that there is quite a difference between an in-process and out of process components. Some things you can share within your process but you can’t share them outside of your process. You don’t need to marshal data with in-process calls and you do out of process. There isn’t normally a change of context (for example, user account) in an in-process call but there often is with an out-of-process call. Also, a client would reasonably expect to see the interfaces supplied by your DLL and so VB6.exe suddenly grew a lot of new interfaces when it was hosting your component.

This was all very clever and a little bit fragile. As MTS grew up and became COM+, it stopped working because the interrelationships between the component and the host became much more complex. You couldn’t always use the IDE to debug hosted components any more. If you want to know how it was done under IIS4 then there is still a KB - http://support.microsoft.com/default.aspx?scid=kb;en-us;299633

Leave Comment:

Name:

Email: (not shown)

Message: (Required)

Math Question: 50 + 87 = ? followed by the letter: C