Duktape Debug Protocol

Author: Dave
Date: 07.17.15 - 6:06pm

Ok this post is going to have a very small audience i probably shouldnt even bother..but this is for my own reference as much as it is for sharing..

I am working with the DukTape binary debugging protocol.

Its a very tight setup that has been optimized to work on even embedded targets with low memory constraints and slow transport mediums such as serial lines.

So the protocol is compact and byte efficient.

Since I am working on windows with direct callbacks between the engine (C dll) and the interface executable (vb6 exe)..I cheated a little and have removed a couple of the byte packing features that would lead me to having to do extra processing. (vb6 does not have bit shift operators)

The following commit has a very simple ifdef added to disable the packing (DUK_DBG_USE_PACKED_PROTOCOL)

The raw binary protocol works through a read/write mechanism. In the example code they proxy this over a network connection. In my implementation its a direct callback into my VB6 code.

Another thing to describe of the protocol, is that since it supports memory constrained devices, it reads and writes portions of requests as needed.

If this section of code needs a byte flag..it reads one byte. If that section of code is emitting a type specifier that is 4 bytes, it writes exactly 4 bytes. As it gets to different sections of code it will emit/read what it requires at that exact point in time. Once it is finished with its response it will emit a single 00 byte to mark end of message (EOM).

This prevents the need to have to have memory available for an entire message packet all at once. (imagine a large string being send over serial from an arduino)

As an consumer of this protocol, this is what you have to work with. for sanity the way I chose to handle this, was to accept whatever micro "packets" it sent me, reassembling them and storing them until I receive the final EOM marker. Then I parse the complete message .

Similarily, when I have a message to send duktape, I created a writebuffer class. So I generate the entire request to send, put it in a buffer and let duktape sip from it as it wishs.

Both of these classes have internal pointers to track data reads so I dont have to. When duktape sends a message for example, I can call .ReadInt; .ReadInt; .ReadString and it will keep grabbing the next value from the buffer for me. They also understand any internal prefixes duktape sends for type encoding.

Ok..soo were not even to the main point of this post yet! thats just the warm up.. lets look at a GET_VAR request. Here I am requesting the value of js variable v1 (remember I disabled the packed protocol for replies)

request var v1 value
REQ   EOM
--------------------------------------------
000000   01 9A 11 00 00 00 02 76 31 00      .......v1.       
         A  |  C  |           E     F
            B     D
                               
A = DUK_DBG_MARKER_REQUEST
B = encoded DUK_DBG_CMD_GETVAR (80+1a)
C = string marker
D = 32bit str len
E = raw string
F = EOM

response
REP   EOM
--------------------------------------------
000000   02 10 00 00 00 01 1A 3F F0 00 00 00 00 00 00 00     
         A  |  C           |  E                       |
            B              D                          F                                  
                           
A = DBG_MARKER_REPLY
B = type int	
C = 32BIT value 1 (found)    
D = varType (double)                               
E = 8 byte network endian double value for 1
F = EOM

Ok so now you have the basic background..read debugger.rst you are going to need it but it does a really good job. also you can use ajs_debugger.c for reference.

So I am working in a single threaded VB6 gui. When a read request comes in and I do not have any data for it, I go into a while not readyToReturn: doevents loop. This blocks the thread within the read request and still lets the GUI respond to other events like button clicks etc. The UI is still responsive.

For most things this is fine..user clicks toolbar to single step, I generate the Step_Into message packet, set readyToReturn = true. The toolbar_Click event handler returns, and vb resumes execution back into the readdata while loop, which now exits and duktape starts sipping off the buffer. Once the buffer is empty, we end up back in the while wait loop.

See reference implementation here

Now lets imagine we need to make a request for a variable value when the mouse dwells over a variable name in our scintilla control. (fancy I know!)

You might not realize it yet..but we now we have a problem.

We need our data synchronously. Our call stack looks like this (assume button click triggered even for clarity)

RequestVariableValue
command1_click
[doevents native code from runtime]
DukTape_Read_CallBack (where our blocking call is)

Through the conventional mechanism we can never receive our variable value where we want it because the blocking call in the read call back has to release to process the packet which will only happen after our command1_click event returns....

One thought I had, was to make the request, then start a timer to keep checking a global flag to see if the value had been retrieved yet or not, then let the timer event do the actual tooltip display with the variable value. This would work but seemed confusing and scattered logic.

After some thought I came up with an elegant way to do it that works and lets me receive the value immediately in a synchronous fashion.

consider the following:

Function SyncronousGetVariableValue(name As String) As CVariable
    Set VarReturn = New CVariable
    VarReturn.name = name
    LastCommand = dc_GetVar
    replyReceived = False 'build GET_VAR packet minus DUK_DBG_MARKER_REQUEST, DUK_DBG_CMD_GETVAR
    RespBuffer.ConstructMessage dc_GetVar, name, True   
    DukOp opd_dbgManuallyTriggerGetVar, ActiveDebuggerClass.Context '
    Set SyncronousGetVariableValue = VarReturn
End Function

Function ConstructMessage(
                d As Debug_Commands, 
                Optional arg1, 
                Optional isManualCall As Boolean = False
) As Boolean
    
    PurgeBuffer
    ...
    
    If d = dc_GetVar Then
        'REQ <int: 0x1a> <str: varname> EOM
        If Not isManualCall Then
            AddByte DUK_DBG_MARKER_REQUEST
            AddByte &H80 + &H1A 'DUK_DBG_CMD_GETVAR
        Else
            'opd_dbgManuallyTriggerGetVar support
        End If
        AddString CStr(arg1)
        AddByte DUK_DBG_MARKER_EOM
        ConstructMessage = True
        Exit Function
    End If
   
End Function

int __stdcall DukOp(int operation, duk_context *ctx, int arg1, char* arg2){
#pragma EXPORT

	switch(operation){
	   ...
	   case opd_dbgManuallyTriggerGetVar:
	          ManuallyTriggerGetVar(ctx);
		  return 0;




void ManuallyTriggerGetVar(duk_context* ctx){
	duk_hthread *thr = (duk_hthread *)ctx;
	duk_heap *heap;

	DUK_ASSERT(thr != NULL);
	heap = thr->heap;
	DUK_ASSERT(heap != NULL);
	DUK_UNREF(ctx);

	duk__debug_handle_get_var(thr, heap);

/*
this direct call bypasses following code:

DUK_LOCAL void duk__debug_process_message(duk_hthread *thr) {
	...
	x = duk_debug_read_byte(thr);
	switch (x) {
	case DUK_DBG_MARKER_REQUEST: {
		cmd = duk_debug_read_int(thr);
		switch (cmd) {
                ...
                    case DUK_DBG_CMD_GETVAR: {
			duk__debug_handle_get_var(thr, heap);
			break;
*/

}

(VarReturn still has to be a private module level variable to work across the callback)

So heres is the fancy part ( which could totally break in teh future but they already warned us the whole protocol could change so..)

We cant return back to my DukTape_Read_CallBack..but can we initiate the request / reply mechanism to get the data we want from our current stack position?

The answer is yes!

Since we know that duktape reads values from the buffer as it requires them..we can call into the internal message processing code directly at a suitable point.

In the example protocol message shown above, the first two bytes 01 9A are read individually. One place to say hey its a request, and 9a for ok its a GETVar request.

At this point duktape calls duk__debug_handle_get_var(duk_hthread *thr, duk_heap *heap) and the reading of the request buffer keeps occuring as normal, followed by its normal response.

So for us to call into this specific spot which is technically mid protocol parse..all we have to do is construct our Get_VAR request buffer, absent the first two bytes and it will process as normal.

By the time our custom DukOp opd_dbgManuallyTriggerGetVar returns, the entire read/write cycle has already completed and our module level VarReturn variable has been set.

This is a pretty unique solution and is working like a charm. Complex, yes..a hack..yes..but for the envirnoment I am working with it was the simpliest solution (technically not logically) and appears stable.

If I didnt really want the ability to do syncronous data calls in other parts of the code I wouldnt have bothered, but I think I will need this in several other places as well.

Phew.. that made my brain hurt!

Comments: (1)

On 07.19.15 - 8:24am Dave wrote:
actually i just had another realization that makes life simpler. Instead of creating a custom protocol packet missing the first two bytes so I can call one of the specific internal functions directly..just calling the top level duk__debug_process_message with a full protocol packet ready to go, does the exact same thing, but with less "special conditions" and care in processing..You can see the simplification in this commit

So long story short..I needed a way for the user to initiate debugger read request and response cycle so that from the point of view of the user, the data was directly available when the trigger returns.

Still sounds complex..but whatever it works.

Leave Comment:

Name:

Email: (not shown)

Message: (Required)

Math Question: 32 + 81 = ? followed by the letter: U