Duktape Debug Protocol


Author: Dave
Date: 07.17.15 - 6:06pm



Ok this post is going to have a very small audience i probably shouldnt even bother..but this is for my own reference as much as it is for sharing..

I am working with the DukTape binary debugging protocol.

Its a very tight setup that has been optimized to work on even embedded targets with low memory constraints and slow transport mediums such as serial lines.

So the protocol is compact and byte efficient.

Since I am working on windows with direct callbacks between the engine (C dll) and the interface executable (vb6 exe)..I cheated a little and have removed a couple of the byte packing features that would lead me to having to do extra processing. (vb6 does not have bit shift operators)

The following commit has a very simple ifdef added to disable the packing (DUK_DBG_USE_PACKED_PROTOCOL)

The raw binary protocol works through a read/write mechanism. In the example code they proxy this over a network connection. In my implementation its a direct callback into my VB6 code.

Another thing to describe of the protocol, is that since it supports memory constrained devices, it reads and writes portions of requests as needed.

If this section of code needs a byte flag..it reads one byte. If that section of code is emitting a type specifier that is 4 bytes, it writes exactly 4 bytes. As it gets to different sections of code it will emit/read what it requires at that exact point in time. Once it is finished with its response it will emit a single 00 byte to mark end of message (EOM).

This prevents the need to have to have memory available for an entire message packet all at once. (imagine a large string being send over serial from an arduino)

As an consumer of this protocol, this is what you have to work with. for sanity the way I chose to handle this, was to accept whatever micro "packets" it sent me, reassembling them and storing them until I receive the final EOM marker. Then I parse the complete message .

Similarily, when I have a message to send duktape, I created a writebuffer class. So I generate the entire request to send, put it in a buffer and let duktape sip from it as it wishs.

Both of these classes have internal pointers to track data reads so I dont have to. When duktape sends a message for example, I can call .ReadInt; .ReadInt; .ReadString and it will keep grabbing the next value from the buffer for me. They also understand any internal prefixes duktape sends for type encoding.

Ok..soo were not even to the main point of this post yet! thats just the warm up.. lets look at a GET_VAR request. Here I am requesting the value of js variable v1 (remember I disabled the packed protocol for replies)

request var v1 value
REQ   EOM
--------------------------------------------
000000   01 9A 11 00 00 00 02 76 31 00      .......v1.       
         A  |  C  |           E     F
            B     D
                               
A = DUK_DBG_MARKER_REQUEST
B = encoded DUK_DBG_CMD_GETVAR (80+1a)
C = string marker
D = 32bit str len
E = raw string
F = EOM

response
REP   EOM
--------------------------------------------
000000   02 10 00 00 00 01 1A 3F F0 00 00 00 00 00 00 00     
         A  |  C           |  E                       |
            B              D                          F                                  
                           
A = DBG_MARKER_REPLY
B = type int	
C = 32BIT value 1 (found)    
D = varType (double)                               
E = 8 byte network endian double value for 1
F = EOM  
Ok so now you have the basic background..read debugger.rst you are going to need it but it does a really good job. also you can use ajs_debugger.c for reference.

So I am working in a single threaded VB6 gui. When a read request comes in and I do not have any data for it, I go into a while not readyToReturn: doevents loop. This blocks the thread within the read request and still lets the GUI respond to other events like button clicks etc. The UI is still responsive.

For most things this is fine..user clicks toolbar to single step, I generate the Step_Into message packet, set readyToReturn = true. The toolbar_Click event handler returns, and vb resumes execution back into the readdata while loop, which now exits and duktape starts sipping off the buffer. Once the buffer is empty, we end up back in the while wait loop.

See reference implementation here

Now lets imagine we need to make a request for a variable value when the mouse dwells over a variable name in our scintilla control. (fancy I know!)

You might not realize it yet..but we now we have a problem.

We need our data synchronously. Our call stack looks like this (assume button click triggered even for clarity)

RequestVariableValue
command1_click
[doevents native code from runtime]
DukTape_Read_CallBack (where our blocking call is)
Through the conventional mechanism we can never receive our variable value where we want it because the blocking call in the read call back has to release to process the packet which will only happen after our command1_click event returns....

One thought I had, was to make the request, then start a timer to keep checking a global flag to see if the value had been retrieved yet or not, then let the timer event do the actual tooltip display with the variable value. This would work but seemed confusing and scattered logic.

After some thought I came up with an elegant way to do it that works and lets me receive the value immediately in a synchronous fashion.

consider the following:

Function SyncronousGetVariableValue(name As String) As CVariable
    Set VarReturn = New CVariable
    VarReturn.name = name
    LastCommand = dc_GetVar
    replyReceived = False 'build GET_VAR packet minus DUK_DBG_MARKER_REQUEST, DUK_DBG_CMD_GETVAR
    RespBuffer.ConstructMessage dc_GetVar, name, True   
    DukOp opd_dbgManuallyTriggerGetVar, ActiveDebuggerClass.Context '
    Set SyncronousGetVariableValue = VarReturn
End Function


Function ConstructMessage(
                d As Debug_Commands, 
                Optional arg1, 
                Optional isManualCall As Boolean = False
) As Boolean
    
    PurgeBuffer
    ...
    
    If d = dc_GetVar Then
        'REQ <int: 0x1a> <str: varname> EOM
        If Not isManualCall Then
            AddByte DUK_DBG_MARKER_REQUEST
            AddByte &H80 + &H1A 'DUK_DBG_CMD_GETVAR
        Else
            'opd_dbgManuallyTriggerGetVar support
        End If
        AddString CStr(arg1)
        AddByte DUK_DBG_MARKER_EOM
        ConstructMessage = True
        Exit Function
    End If
   
End Function


int __stdcall DukOp(int operation, duk_context *ctx, int arg1, char* arg2){
#pragma EXPORT

	switch(operation){
	   ...
	   case opd_dbgManuallyTriggerGetVar:
	          ManuallyTriggerGetVar(ctx);
		  return 0;


void ManuallyTriggerGetVar(duk_context* ctx){ duk_hthread *thr = (duk_hthread *)ctx; duk_heap *heap; DUK_ASSERT(thr != NULL); heap = thr->heap; DUK_ASSERT(heap != NULL); DUK_UNREF(ctx); duk__debug_handle_get_var(thr, heap); /* this direct call bypasses following code: DUK_LOCAL void duk__debug_process_message(duk_hthread *thr) { ... x = duk_debug_read_byte(thr); switch (x) { case DUK_DBG_MARKER_REQUEST: { cmd = duk_debug_read_int(thr); switch (cmd) { ... case DUK_DBG_CMD_GETVAR: { duk__debug_handle_get_var(thr, heap); break; */ }


(VarReturn still has to be a private module level variable to work across the callback)

So heres is the fancy part ( which could totally break in teh future but they already warned us the whole protocol could change so..)

We cant return back to my DukTape_Read_CallBack..but can we initiate the request / reply mechanism to get the data we want from our current stack position?

The answer is yes!

Since we know that duktape reads values from the buffer as it requires them..we can call into the internal message processing code directly at a suitable point.

In the example protocol message shown above, the first two bytes 01 9A are read individually. One place to say hey its a request, and 9a for ok its a GETVar request.

At this point duktape calls duk__debug_handle_get_var(duk_hthread *thr, duk_heap *heap) and the reading of the request buffer keeps occuring as normal, followed by its normal response.

So for us to call into this specific spot which is technically mid protocol parse..all we have to do is construct our Get_VAR request buffer, absent the first two bytes and it will process as normal.

By the time our custom DukOp opd_dbgManuallyTriggerGetVar returns, the entire read/write cycle has already completed and our module level VarReturn variable has been set.

This is a pretty unique solution and is working like a charm. Complex, yes..a hack..yes..but for the envirnoment I am working with it was the simpliest solution (technically not logically) and appears stable.

If I didnt really want the ability to do syncronous data calls in other parts of the code I wouldnt have bothered, but I think I will need this in several other places as well.

Phew.. that made my brain hurt!






Comments: (1)

On 07.19.15 - 8:24am Dave wrote:
actually i just had another realization that makes life simpler. Instead of creating a custom protocol packet missing the first two bytes so I can call one of the specific internal functions directly..just calling the top level duk__debug_process_message with a full protocol packet ready to go, does the exact same thing, but with less "special conditions" and care in processing..You can see the simplification in this commit

So long story short..I needed a way for the user to initiate debugger read request and response cycle so that from the point of view of the user, the data was directly available when the trigger returns.

Still sounds complex..but whatever it works.

 
Leave Comment:
Name:
Email: (not shown)
Message: (Required)
Math Question: 20 + 40 = ? followed by the letter: K 



About Me
More Blogs
Main Site
Posts: (year)
2022 (4)
     More VB6 - C data passing
     Vb6 Asm listing
     Byte Array C to VB6
     Planet Source Code DVDs
2021 (2)
     Obscure VB
     VB6 IDE SP6
2020 (5)
     NTFileSize
     BSTR from C Dll to VB
     Cpp Memory Manipulation
     ActiveX Binary Compatability
     C libs for VB6
2019 (5)
     Console tricks
     FireFox temp dir
     OCX License
     Extract substring
     VB6 Console Apps
2018 (6)
     VB6 UDTs
     VB6 Debugger View As Hex tooltips
     VB6 - C Share registry data
     VB6 Addin Missing Menus
     VB6 Class Init Params
     VB6 isIn function
2017 (6)
     Python and VB6
     Python pros and cons
     download web Dir
     vc rand in python
     VB6 Language Enhancement
     Register .NET as COM
2016 (22)
     VB6 CDECL
     UDT Tricks pt2
     Remote Data Extraction
     Collection Extender
     VB6 FindResource
     CDO.Message
     DirList Single Click
     Reset CheckPoint VPN Policy
     VB6 BSTR Oddities Explained
     SafeArrays in C
     BSTR and Variant in C++
     Property let optional args
     Misc Libs
     Enum Named Pipes
     Vb6 Collection in C++
     VB6 Overloaded Methods
     EXPORT FUNCDNAME Warning
     VB6 Syncronous Socket
     Simple IPC
     VB6 Auto Resize Form Elements
     Mach3 Automation
     Exit For in While
2015 (16)
     C# self register ocx
     VB6 Class Method Pointers
     JS Debugger
     Duktape Debug Protocol
     QtScript 4 VB
     Vb6 Named Args
     vb6 Addin Part 2
     VB6 Addin vrs Toolbars
     OpenFile Dialog MultiSelect
     Duktape Example
     DukTape JS
     VB6 Unsigned
     .Net version
     TitleBar Height
     .NET again
     VB6 Self Register OCXs
2014 (25)
     Query Last 12 Mos
     Progid from Interface ID
     VB6 to C Array Examples
     Human Readable Variant Type
     ScriptBasic COM Integration
     CodeView Addin
     ScriptBasic - Part 2
     Script Env
     MSCOMCTL Win7 Error
     printf override
     History Combo
     Disable IE
     API Hooking in VB6
     Addin Hook Events
     FastBuild Addin
     VB6 MemoryWindow
     Link C Obj Files into VB6
     Vb6 Standard Dlls
     CStr for Pascal
     Lazarus Review
     asprintf for VS
     VB6 GlobalMultiUse
     Scintilla in VB6
     Dynamic Highlight
     WinVerifyTrust, CryptMsgGetParam VB6
2013 (4)
     MS GLEE Graphing
     printf for VB6
     C# App Config
     Tero DES C# Test
2012 (10)
     VC 2008 Bit Fields
     Speed trap
     C# Db Class Generator
     VB6 vrs .NET (again)
     FireFox Whois Extension
     git and vb6
     Code Additions
     Compiled date to string
     C# ListView Sorter
     VB6 Wish List
2011 (7)
     C# Process Injection
     CAPTCHA Bots
     C# PE Offset Calculator
     VB6 Async Download
     Show Desktop
     coding philosophy
     Code release
2010 (11)
     Dll Not Found in IDE
     Advanced MSScript Control
     random tip
     Clipart / Vector Art
     VB6 Callback from C#
     Binary data from VB6 to C#
     CSharp and MsScriptControl
     HexDumper functions
     Js Beautify From VB6 or C#
     vb6 FormPos
     Inline Asm w VB6
2009 (3)
     The .NET Fiasco
     One rub on computers
     Universal extractor