VB6 BSTR Oddities Explained


Author: Dave
Date: 04.07.16 - 2:44am



tldr; scroll down to the big blue words...

Ok, so working with strings between VB6 and C.

I have actually had some misconceptions about this for quite a long time. Generally what I have done is to pass them byVal in the api declaration which I was told..converts then to an ansi C string to pass to C, and then converts them back to unicode on the way back so you can use them as buffers to fill in. You also have to pass in the string buffer size as an argument and allocate the string length buffer in vb.

ex:
'c prototype: void __stdcall method(char* str, int sLen)
private sub method lib "mydll"(byval str as string, byval sLen as long)

dim s as string 
s = space(100)
method(s,100)
Now having to pass in all the sLen arguments all the time is a pain, so I start looking around and find an example of passing the string in byref
'c prototype: void __stdcall method(BSTR* str)
private sub method lib "mydll"(byref str as string)
So I noticed something fishy about this when working with it in C.

void __stdcall chainit(BSTR* _key, BSTR* _nonce, uint32_t count)
{
	if(_nonce != 0){
		uint32_t nLen = SysStringByteLen(*_nonce);
		if(nLen > 0){
			if(nLen > 8) nLen = 8;
			memcpy(nonce,(uint8_t*)*_nonce,nLen);
		}
	}


Do you see it? (uint8_t*)*_nonce

To work with it, we cast it as an ascii type, even through it should be a wide char BSTR, we passed it by ref.

In VB6 our string is true unicode. But even though we are passing it to C byref and hoping to get a bstr..and we do actually get a bstr, you can use SysStringByteLen(*str) on it..but its no longer wide char, even though its still in a BSTR.

See the following:



Ok..now if you scroll the C memory window up to the next line, guess what you see..the BSTR length prefix. So what the runtime did was convert our Wide string to an ascii string even though it is still housed in a BSTR container. even through we passed it by ref. Now if you want a legit wide bstr unaltered, you can get it:
'c prototype: void __stdcall method(BSTR* str)
private sub method lib "mydll"(byref str as long)
method(strptr("my string"))
thats another story though..

so the point I am going for..in our first example the runtime was also converting our wide string to ascii for us..but here is hte point I never realize..and I dont think many people do...its the same mechanism at play..scroll up a line in the memory window in the first byval example and guess what you find. the BSTR length prefix.

So all of this time we have been littering our code with all of these buffer length arguments...not necessary.

In the api examples, it was necessary, because thats old C code written for C. But if you are writing your own C dll for VB, you dont have to follow old C conventions. You can use SysStringByteLen even on a string passed in byval

/*
   Private Declare Sub test Lib "libchacha" (ByVal key As String)
*/
void __stdcall test(BSTR _key)
{
	char buf[55];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen(_key);
		sprintf(buf,"BSTR Len: %d",kLen);
		MessageBox(0,buf,"",0);
	}
}


thus by logical extension..the following also works:

void __stdcall test(char* _key)
{
	char buf[55];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen((BSTR)_key);
		sprintf(buf,"BSTR Len: %d",kLen);
		MessageBox(0,buf,"",0);
	}
}


note that this only works because of how specifically the vb6 runtime handles its api declares..this is completely invalid for any other C coding!

I feel like I have been kept in the dark as to teh true nature of how vb6 handles strings to api methods and thats why i always had some confusion on this. When things dont quite gel, its a good sign you are missing part of the story. Here its because they are playing party tricks on us.

Moral of the story..whether you are passing byval or byref, and defining as char* or BSTR in C, you are always getting a BSTR in reality, but never a wide char BSTR, always one of these converted ones. It will also always be converted back to wide by the runtime on return.

Also that bit about only being able to change strings passed in byval, false.

/*
Private Declare Sub test2 Lib "libchacha" (ByRef key As String)

    Dim a As String
    a = "my string!"
    test2 a
    MsgBox a

*/

//byref char** or BSTR*
void __stdcall test2(BSTR* _key)
{
	char buf[1024];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen(*_key);
		if(kLen > 1000) return; //fu
		sprintf(buf,"BSTR '%s' Len: %d", *_key, kLen);
		MessageBox(0,buf,"",0);
		char* b = (char*)*_key;
		b[kLen-1] = '?';
	}
}


One other thing I never really got until now. A BSTR pointer is pretty weird. consider it as a struct:
struct BSTR{
  uint32_t size,
  WCHAR buf[size],
};
When you pass a BSTR in, the pointer actually points to the buf element already inside of it. They did this so that it was compatiable with things that expect wchar buffers. When they always say that a BSTR has a length prefix..they really arent kidding. The size is actually before the pointer you are passed. I dont know of any other structures like this. Again its a trick they played for compatibility.

And you wonder why C developers find COM confusing :)




Comments: (1)

On 05.01.16 - 8:44am Dave wrote:
one more note, when passing a string buffer to C from VB, do NOT use lenb() to calculate the buffer size. lenb returns the number of bytes in the unicode string, remember that the string will be converted from wide to ansi by the runtime! You will in effect be passing size*2 to your code C would can easily result in a buffer overflow.

 
Leave Comment:
Name:
Email: (not shown)
Message: (Required)
Math Question: 85 + 15 = ? followed by the letter: G 



About Me
More Blogs
Main Site
Posts: (year)
2022 (4)
     More VB6 - C data passing
     Vb6 Asm listing
     Byte Array C to VB6
     Planet Source Code DVDs
2021 (2)
     Obscure VB
     VB6 IDE SP6
2020 (5)
     NTFileSize
     BSTR from C Dll to VB
     Cpp Memory Manipulation
     ActiveX Binary Compatability
     C libs for VB6
2019 (5)
     Console tricks
     FireFox temp dir
     OCX License
     Extract substring
     VB6 Console Apps
2018 (6)
     VB6 UDTs
     VB6 Debugger View As Hex tooltips
     VB6 - C Share registry data
     VB6 Addin Missing Menus
     VB6 Class Init Params
     VB6 isIn function
2017 (6)
     Python and VB6
     Python pros and cons
     download web Dir
     vc rand in python
     VB6 Language Enhancement
     Register .NET as COM
2016 (22)
     VB6 CDECL
     UDT Tricks pt2
     Remote Data Extraction
     Collection Extender
     VB6 FindResource
     CDO.Message
     DirList Single Click
     Reset CheckPoint VPN Policy
     VB6 BSTR Oddities Explained
     SafeArrays in C
     BSTR and Variant in C++
     Property let optional args
     Misc Libs
     Enum Named Pipes
     Vb6 Collection in C++
     VB6 Overloaded Methods
     EXPORT FUNCDNAME Warning
     VB6 Syncronous Socket
     Simple IPC
     VB6 Auto Resize Form Elements
     Mach3 Automation
     Exit For in While
2015 (16)
     C# self register ocx
     VB6 Class Method Pointers
     JS Debugger
     Duktape Debug Protocol
     QtScript 4 VB
     Vb6 Named Args
     vb6 Addin Part 2
     VB6 Addin vrs Toolbars
     OpenFile Dialog MultiSelect
     Duktape Example
     DukTape JS
     VB6 Unsigned
     .Net version
     TitleBar Height
     .NET again
     VB6 Self Register OCXs
2014 (25)
     Query Last 12 Mos
     Progid from Interface ID
     VB6 to C Array Examples
     Human Readable Variant Type
     ScriptBasic COM Integration
     CodeView Addin
     ScriptBasic - Part 2
     Script Env
     MSCOMCTL Win7 Error
     printf override
     History Combo
     Disable IE
     API Hooking in VB6
     Addin Hook Events
     FastBuild Addin
     VB6 MemoryWindow
     Link C Obj Files into VB6
     Vb6 Standard Dlls
     CStr for Pascal
     Lazarus Review
     asprintf for VS
     VB6 GlobalMultiUse
     Scintilla in VB6
     Dynamic Highlight
     WinVerifyTrust, CryptMsgGetParam VB6
2013 (4)
     MS GLEE Graphing
     printf for VB6
     C# App Config
     Tero DES C# Test
2012 (10)
     VC 2008 Bit Fields
     Speed trap
     C# Db Class Generator
     VB6 vrs .NET (again)
     FireFox Whois Extension
     git and vb6
     Code Additions
     Compiled date to string
     C# ListView Sorter
     VB6 Wish List
2011 (7)
     C# Process Injection
     CAPTCHA Bots
     C# PE Offset Calculator
     VB6 Async Download
     Show Desktop
     coding philosophy
     Code release
2010 (11)
     Dll Not Found in IDE
     Advanced MSScript Control
     random tip
     Clipart / Vector Art
     VB6 Callback from C#
     Binary data from VB6 to C#
     CSharp and MsScriptControl
     HexDumper functions
     Js Beautify From VB6 or C#
     vb6 FormPos
     Inline Asm w VB6
2009 (3)
     The .NET Fiasco
     One rub on computers
     Universal extractor