VB6 BSTR Oddities Explained

Author: Dave
Date: 04.07.16 - 2:44am

tldr; scroll down to the big blue words...

Ok, so working with strings between VB6 and C.

I have actually had some misconceptions about this for quite a long time. Generally what I have done is to pass them byVal in the api declaration which I was told..converts then to an ansi C string to pass to C, and then converts them back to unicode on the way back so you can use them as buffers to fill in. You also have to pass in the string buffer size as an argument and allocate the string length buffer in vb.

ex:

'c prototype: void __stdcall method(char* str, int sLen)
private sub method lib "mydll"(byval str as string, byval sLen as long)

dim s as string 
s = space(100)
method(s,100)

Now having to pass in all the sLen arguments all the time is a pain, so I start looking around and find an example of passing the string in byref

'c prototype: void __stdcall method(BSTR* str)
private sub method lib "mydll"(byref str as string)

So I noticed something fishy about this when working with it in C.

void __stdcall chainit(BSTR* _key, BSTR* _nonce, uint32_t count)
{
	if(_nonce != 0){
		uint32_t nLen = SysStringByteLen(*_nonce);
		if(nLen > 0){
			if(nLen > 8) nLen = 8;
			memcpy(nonce,(uint8_t*)*_nonce,nLen);
		}
	}

Do you see it? (uint8_t*)*_nonce

To work with it, we cast it as an ascii type, even through it should be a wide char BSTR, we passed it by ref.

In VB6 our string is true unicode. But even though we are passing it to C byref and hoping to get a bstr..and we do actually get a bstr, you can use SysStringByteLen(*str) on it..but its no longer wide char, even though its still in a BSTR.

See the following:

Ok..now if you scroll the C memory window up to the next line, guess what you see..the BSTR length prefix. So what the runtime did was convert our Wide string to an ascii string even though it is still housed in a BSTR container. even through we passed it by ref. Now if you want a legit wide bstr unaltered, you can get it:

'c prototype: void __stdcall method(BSTR* str)
private sub method lib "mydll"(byref str as long)
method(strptr("my string"))

thats another story though..

so the point I am going for..in our first example the runtime was also converting our wide string to ascii for us..but here is hte point I never realize..and I dont think many people do...its the same mechanism at play..scroll up a line in the memory window in the first byval example and guess what you find. the BSTR length prefix.

So all of this time we have been littering our code with all of these buffer length arguments...not necessary.

In the api examples, it was necessary, because thats old C code written for C. But if you are writing your own C dll for VB, you dont have to follow old C conventions. You can use SysStringByteLen even on a string passed in byval

/*
   Private Declare Sub test Lib "libchacha" (ByVal key As String)
*/
void __stdcall test(BSTR _key)
{
	char buf[55];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen(_key);
		sprintf(buf,"BSTR Len: %d",kLen);
		MessageBox(0,buf,"",0);
	}
}

thus by logical extension..the following also works:

void __stdcall test(char* _key)
{
	char buf[55];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen((BSTR)_key);
		sprintf(buf,"BSTR Len: %d",kLen);
		MessageBox(0,buf,"",0);
	}
}

note that this only works because of how specifically the vb6 runtime handles its api declares..this is completely invalid for any other C coding!

I feel like I have been kept in the dark as to teh true nature of how vb6 handles strings to api methods and thats why i always had some confusion on this. When things dont quite gel, its a good sign you are missing part of the story. Here its because they are playing party tricks on us.

Moral of the story..whether you are passing byval or byref, and defining as char* or BSTR in C, you are always getting a BSTR in reality, but never a wide char BSTR, always one of these converted ones. It will also always be converted back to wide by the runtime on return.

Also that bit about only being able to change strings passed in byval, false.

/*
Private Declare Sub test2 Lib "libchacha" (ByRef key As String)

    Dim a As String
    a = "my string!"
    test2 a
    MsgBox a

*/

//byref char** or BSTR*
void __stdcall test2(BSTR* _key)
{
	char buf[1024];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen(*_key);
		if(kLen > 1000) return; //fu
		sprintf(buf,"BSTR '%s' Len: %d", *_key, kLen);
		MessageBox(0,buf,"",0);
		char* b = (char*)*_key;
		b[kLen-1] = '?';
	}
}

One other thing I never really got until now. A BSTR pointer is pretty weird. consider it as a struct:

struct BSTR{
  uint32_t size,
  WCHAR buf[size],
};

When you pass a BSTR in, the pointer actually points to the buf element already inside of it. They did this so that it was compatiable with things that expect wchar buffers. When they always say that a BSTR has a length prefix..they really arent kidding. The size is actually before the pointer you are passed. I dont know of any other structures like this. Again its a trick they played for compatibility.

And you wonder why C developers find COM confusing :)

Comments: (1)

On 05.01.16 - 8:44am Dave wrote:
one more note, when passing a string buffer to C from VB, do NOT use lenb() to calculate the buffer size. lenb returns the number of bytes in the unicode string, remember that the string will be converted from wide to ansi by the runtime! You will in effect be passing size*2 to your code C would can easily result in a buffer overflow.

Leave Comment:

Name:

Email: (not shown)

Message: (Required)

Math Question: 29 + 8 = ? followed by the letter: V