Duktape Debug Protocol


Author: Dave
Date: 07.17.15 - 6:06pm



Ok this post is going to have a very small audience i probably shouldnt even bother..but this is for my own reference as much as it is for sharing..

I am working with the DukTape binary debugging protocol.

Its a very tight setup that has been optimized to work on even embedded targets with low memory constraints and slow transport mediums such as serial lines.

So the protocol is compact and byte efficient.

Since I am working on windows with direct callbacks between the engine (C dll) and the interface executable (vb6 exe)..I cheated a little and have removed a couple of the byte packing features that would lead me to having to do extra processing. (vb6 does not have bit shift operators)

The following commit has a very simple ifdef added to disable the packing (DUK_DBG_USE_PACKED_PROTOCOL)

The raw binary protocol works through a read/write mechanism. In the example code they proxy this over a network connection. In my implementation its a direct callback into my VB6 code.

Another thing to describe of the protocol, is that since it supports memory constrained devices, it reads and writes portions of requests as needed.

If this section of code needs a byte flag..it reads one byte. If that section of code is emitting a type specifier that is 4 bytes, it writes exactly 4 bytes. As it gets to different sections of code it will emit/read what it requires at that exact point in time. Once it is finished with its response it will emit a single 00 byte to mark end of message (EOM).

This prevents the need to have to have memory available for an entire message packet all at once. (imagine a large string being send over serial from an arduino)

As an consumer of this protocol, this is what you have to work with. for sanity the way I chose to handle this, was to accept whatever micro "packets" it sent me, reassembling them and storing them until I receive the final EOM marker. Then I parse the complete message .

Similarily, when I have a message to send duktape, I created a writebuffer class. So I generate the entire request to send, put it in a buffer and let duktape sip from it as it wishs.

Both of these classes have internal pointers to track data reads so I dont have to. When duktape sends a message for example, I can call .ReadInt; .ReadInt; .ReadString and it will keep grabbing the next value from the buffer for me. They also understand any internal prefixes duktape sends for type encoding.

Ok..soo were not even to the main point of this post yet! thats just the warm up.. lets look at a GET_VAR request. Here I am requesting the value of js variable v1 (remember I disabled the packed protocol for replies)

request var v1 value
REQ   EOM
--------------------------------------------
000000   01 9A 11 00 00 00 02 76 31 00      .......v1.       
         A  |  C  |           E     F
            B     D
                               
A = DUK_DBG_MARKER_REQUEST
B = encoded DUK_DBG_CMD_GETVAR (80+1a)
C = string marker
D = 32bit str len
E = raw string
F = EOM

response
REP   EOM
--------------------------------------------
000000   02 10 00 00 00 01 1A 3F F0 00 00 00 00 00 00 00     
         A  |  C           |  E                       |
            B              D                          F                                  
                           
A = DBG_MARKER_REPLY
B = type int	
C = 32BIT value 1 (found)    
D = varType (double)                               
E = 8 byte network endian double value for 1
F = EOM  
Ok so now you have the basic background..read debugger.rst you are going to need it but it does a really good job. also you can use ajs_debugger.c for reference.

So I am working in a single threaded VB6 gui. When a read request comes in and I do not have any data for it, I go into a while not readyToReturn: doevents loop. This blocks the thread within the read request and still lets the GUI respond to other events like button clicks etc. The UI is still responsive.

For most things this is fine..user clicks toolbar to single step, I generate the Step_Into message packet, set readyToReturn = true. The toolbar_Click event handler returns, and vb resumes execution back into the readdata while loop, which now exits and duktape starts sipping off the buffer. Once the buffer is empty, we end up back in the while wait loop.

See reference implementation here

Now lets imagine we need to make a request for a variable value when the mouse dwells over a variable name in our scintilla control. (fancy I know!)

You might not realize it yet..but we now we have a problem.

We need our data synchronously. Our call stack looks like this (assume button click triggered even for clarity)

RequestVariableValue
command1_click
[doevents native code from runtime]
DukTape_Read_CallBack (where our blocking call is)
Through the conventional mechanism we can never receive our variable value where we want it because the blocking call in the read call back has to release to process the packet which will only happen after our command1_click event returns....

One thought I had, was to make the request, then start a timer to keep checking a global flag to see if the value had been retrieved yet or not, then let the timer event do the actual tooltip display with the variable value. This would work but seemed confusing and scattered logic.

After some thought I came up with an elegant way to do it that works and lets me receive the value immediately in a synchronous fashion.

consider the following:

Function SyncronousGetVariableValue(name As String) As CVariable
    Set VarReturn = New CVariable
    VarReturn.name = name
    LastCommand = dc_GetVar
    replyReceived = False 'build GET_VAR packet minus DUK_DBG_MARKER_REQUEST, DUK_DBG_CMD_GETVAR
    RespBuffer.ConstructMessage dc_GetVar, name, True   
    DukOp opd_dbgManuallyTriggerGetVar, ActiveDebuggerClass.Context '
    Set SyncronousGetVariableValue = VarReturn
End Function


Function ConstructMessage(
                d As Debug_Commands, 
                Optional arg1, 
                Optional isManualCall As Boolean = False
) As Boolean
    
    PurgeBuffer
    ...
    
    If d = dc_GetVar Then
        'REQ <int: 0x1a> <str: varname> EOM
        If Not isManualCall Then
            AddByte DUK_DBG_MARKER_REQUEST
            AddByte &H80 + &H1A 'DUK_DBG_CMD_GETVAR
        Else
            'opd_dbgManuallyTriggerGetVar support
        End If
        AddString CStr(arg1)
        AddByte DUK_DBG_MARKER_EOM
        ConstructMessage = True
        Exit Function
    End If
   
End Function


int __stdcall DukOp(int operation, duk_context *ctx, int arg1, char* arg2){
#pragma EXPORT

	switch(operation){
	   ...
	   case opd_dbgManuallyTriggerGetVar:
	          ManuallyTriggerGetVar(ctx);
		  return 0;


void ManuallyTriggerGetVar(duk_context* ctx){ duk_hthread *thr = (duk_hthread *)ctx; duk_heap *heap; DUK_ASSERT(thr != NULL); heap = thr->heap; DUK_ASSERT(heap != NULL); DUK_UNREF(ctx); duk__debug_handle_get_var(thr, heap); /* this direct call bypasses following code: DUK_LOCAL void duk__debug_process_message(duk_hthread *thr) { ... x = duk_debug_read_byte(thr); switch (x) { case DUK_DBG_MARKER_REQUEST: { cmd = duk_debug_read_int(thr); switch (cmd) { ... case DUK_DBG_CMD_GETVAR: { duk__debug_handle_get_var(thr, heap); break; */ }


(VarReturn still has to be a private module level variable to work across the callback)

So heres is the fancy part ( which could totally break in teh future but they already warned us the whole protocol could change so..)

We cant return back to my DukTape_Read_CallBack..but can we initiate the request / reply mechanism to get the data we want from our current stack position?

The answer is yes!

Since we know that duktape reads values from the buffer as it requires them..we can call into the internal message processing code directly at a suitable point.

In the example protocol message shown above, the first two bytes 01 9A are read individually. One place to say hey its a request, and 9a for ok its a GETVar request.

At this point duktape calls duk__debug_handle_get_var(duk_hthread *thr, duk_heap *heap) and the reading of the request buffer keeps occuring as normal, followed by its normal response.

So for us to call into this specific spot which is technically mid protocol parse..all we have to do is construct our Get_VAR request buffer, absent the first two bytes and it will process as normal.

By the time our custom DukOp opd_dbgManuallyTriggerGetVar returns, the entire read/write cycle has already completed and our module level VarReturn variable has been set.

This is a pretty unique solution and is working like a charm. Complex, yes..a hack..yes..but for the envirnoment I am working with it was the simpliest solution (technically not logically) and appears stable.

If I didnt really want the ability to do syncronous data calls in other parts of the code I wouldnt have bothered, but I think I will need this in several other places as well.

Phew.. that made my brain hurt!






Comments: (1)

On 07.19.15 - 8:24am Dave wrote:
actually i just had another realization that makes life simpler. Instead of creating a custom protocol packet missing the first two bytes so I can call one of the specific internal functions directly..just calling the top level duk__debug_process_message with a full protocol packet ready to go, does the exact same thing, but with less "special conditions" and care in processing..You can see the simplification in this commit

So long story short..I needed a way for the user to initiate debugger read request and response cycle so that from the point of view of the user, the data was directly available when the trigger returns.

Still sounds complex..but whatever it works.

 
Leave Comment:
Name:
Email: (not shown)
Message: (Required)
Math Question: 26 + 95 = ? followed by the letter: U 



About Me
More Blogs
Main Site
Posts:
Console tricks
FireFox temp dir
OCX License
Extract substring
VB6 Console Apps
VB6 UDTs
VB6 Debugger View As Hex tooltips
VB6 - C Share registry data
VB6 Addin Missing Menus
VB6 Class Init Params
VB6 isIn function
Python and VB6
Python pros and cons
download web Dir
vc rand in python
VB6 Language Enhancement
Register .NET as COM
VB6 CDECL
UDT Tricks pt2
Remote Data Extraction
Collection Extender
VB6 FindResource
CDO.Message
DirList Single Click
Reset CheckPoint VPN Policy
VB6 BSTR Oddities Explained
SafeArrays in C
BSTR and Variant in C++
Property let optional args
Misc Libs
Enum Named Pipes
Vb6 Collection in C++
VB6 Overloaded Methods
EXPORT FUNCDNAME Warning
VB6 Syncronous Socket
Simple IPC
VB6 Auto Resize Form Elements
Mach3 Automation
Exit For in While
C# self register ocx
VB6 Class Method Pointers
JS Debugger
Duktape Debug Protocol
QtScript 4 VB
Vb6 Named Args
vb6 Addin Part 2
VB6 Addin vrs Toolbars
OpenFile Dialog MultiSelect
Duktape Example
DukTape JS
VB6 Unsigned
.Net version
TitleBar Height
.NET again
VB6 Self Register OCXs
Query Last 12 Mos
Progid from Interface ID
VB6 to C Array Examples
Human Readable Variant Type
ScriptBasic COM Integration
CodeView Addin
ScriptBasic - Part 2
Script Env
MSCOMCTL Win7 Error
printf override
History Combo
Disable IE
API Hooking in VB6
Addin Hook Events
FastBuild Addin
VB6 MemoryWindow
Link C Obj Files into VB6
Vb6 Standard Dlls
CStr for Pascal
Lazarus Review
asprintf for VS
VB6 GlobalMultiUse
Scintilla in VB6
Dynamic Highlight
WinVerifyTrust, CryptMsgGetParam VB6
MS GLEE Graphing
printf for VB6
C# App Config
Tero DES C# Test
VC 2008 Bit Fields
Speed trap
C# Db Class Generator
VB6 vrs .NET (again)
FireFox Whois Extension
git and vb6
Code Additions
Compiled date to string
C# ListView Sorter
VB6 Wish List
C# Process Injection
CAPTCHA Bots
C# PE Offset Calculator
VB6 Async Download
Show Desktop
coding philosophy
Code release
Dll Not Found in IDE
Advanced MSScript Control
random tip
Clipart / Vector Art
VB6 Callback from C#
Binary data from VB6 to C#
CSharp and MsScriptControl
HexDumper functions
Js Beautify From VB6 or C#
vb6 FormPos
Inline Asm w VB6
The .NET Fiasco
One rub on computers
Universal extractor