VB6 BSTR Oddities Explained


Author: Dave
Date: 04.07.16 - 2:44am



tldr; scroll down to the big blue words...

Ok, so working with strings between VB6 and C.

I have actually had some misconceptions about this for quite a long time. Generally what I have done is to pass them byVal in the api declaration which I was told..converts then to an ansi C string to pass to C, and then converts them back to unicode on the way back so you can use them as buffers to fill in. You also have to pass in the string buffer size as an argument and allocate the string length buffer in vb.

ex:
'c prototype: void __stdcall method(char* str, int sLen)
private sub method lib "mydll"(byval str as string, byval sLen as long)

dim s as string 
s = space(100)
method(s,100)
Now having to pass in all the sLen arguments all the time is a pain, so I start looking around and find an example of passing the string in byref
'c prototype: void __stdcall method(BSTR* str)
private sub method lib "mydll"(byref str as string)
So I noticed something fishy about this when working with it in C.

void __stdcall chainit(BSTR* _key, BSTR* _nonce, uint32_t count)
{
	if(_nonce != 0){
		uint32_t nLen = SysStringByteLen(*_nonce);
		if(nLen > 0){
			if(nLen > 8) nLen = 8;
			memcpy(nonce,(uint8_t*)*_nonce,nLen);
		}
	}


Do you see it? (uint8_t*)*_nonce

To work with it, we cast it as an ascii type, even through it should be a wide char BSTR, we passed it by ref.

In VB6 our string is true unicode. But even though we are passing it to C byref and hoping to get a bstr..and we do actually get a bstr, you can use SysStringByteLen(*str) on it..but its no longer wide char, even though its still in a BSTR.

See the following:



Ok..now if you scroll the C memory window up to the next line, guess what you see..the BSTR length prefix. So what the runtime did was convert our Wide string to an ascii string even though it is still housed in a BSTR container. even through we passed it by ref. Now if you want a legit wide bstr unaltered, you can get it:
'c prototype: void __stdcall method(BSTR* str)
private sub method lib "mydll"(byref str as long)
method(strptr("my string"))
thats another story though..

so the point I am going for..in our first example the runtime was also converting our wide string to ascii for us..but here is hte point I never realize..and I dont think many people do...its the same mechanism at play..scroll up a line in the memory window in the first byval example and guess what you find. the BSTR length prefix.

So all of this time we have been littering our code with all of these buffer length arguments...not necessary.

In the api examples, it was necessary, because thats old C code written for C. But if you are writing your own C dll for VB, you dont have to follow old C conventions. You can use SysStringByteLen even on a string passed in byval

/*
   Private Declare Sub test Lib "libchacha" (ByVal key As String)
*/
void __stdcall test(BSTR _key)
{
	char buf[55];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen(_key);
		sprintf(buf,"BSTR Len: %d",kLen);
		MessageBox(0,buf,"",0);
	}
}


thus by logical extension..the following also works:

void __stdcall test(char* _key)
{
	char buf[55];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen((BSTR)_key);
		sprintf(buf,"BSTR Len: %d",kLen);
		MessageBox(0,buf,"",0);
	}
}


note that this only works because of how specifically the vb6 runtime handles its api declares..this is completely invalid for any other C coding!

I feel like I have been kept in the dark as to teh true nature of how vb6 handles strings to api methods and thats why i always had some confusion on this. When things dont quite gel, its a good sign you are missing part of the story. Here its because they are playing party tricks on us.

Moral of the story..whether you are passing byval or byref, and defining as char* or BSTR in C, you are always getting a BSTR in reality, but never a wide char BSTR, always one of these converted ones. It will also always be converted back to wide by the runtime on return.

Also that bit about only being able to change strings passed in byval, false.

/*
Private Declare Sub test2 Lib "libchacha" (ByRef key As String)

    Dim a As String
    a = "my string!"
    test2 a
    MsgBox a

*/

//byref char** or BSTR*
void __stdcall test2(BSTR* _key)
{
	char buf[1024];
	if(_key != 0){
		uint32_t kLen = SysStringByteLen(*_key);
		if(kLen > 1000) return; //fu
		sprintf(buf,"BSTR '%s' Len: %d", *_key, kLen);
		MessageBox(0,buf,"",0);
		char* b = (char*)*_key;
		b[kLen-1] = '?';
	}
}


One other thing I never really got until now. A BSTR pointer is pretty weird. consider it as a struct:
struct BSTR{
  uint32_t size,
  WCHAR buf[size],
};
When you pass a BSTR in, the pointer actually points to the buf element already inside of it. They did this so that it was compatiable with things that expect wchar buffers. When they always say that a BSTR has a length prefix..they really arent kidding. The size is actually before the pointer you are passed. I dont know of any other structures like this. Again its a trick they played for compatibility.

And you wonder why C developers find COM confusing :)




Comments: (1)

On 05.01.16 - 8:44am Dave wrote:
one more note, when passing a string buffer to C from VB, do NOT use lenb() to calculate the buffer size. lenb returns the number of bytes in the unicode string, remember that the string will be converted from wide to ansi by the runtime! You will in effect be passing size*2 to your code C would can easily result in a buffer overflow.

 
Leave Comment:
Name:
Email: (not shown)
Message: (Required)
Math Question: 25 + 75 = ? followed by the letter: L 



About Me
More Blogs
Main Site
Posts:
VB6 Console Apps
VB6 UDTs
VB6 Debugger View As Hex tooltips
VB6 - C Share registry data
VB6 Addin Missing Menus
VB6 Class Init Params
VB6 isIn function
Python and VB6
Python pros and cons
download web Dir
vc rand in python
VB6 Language Enhancement
Register .NET as COM
VB6 CDECL
UDT Tricks pt2
Remote Data Extraction
Collection Extender
VB6 FindResource
CDO.Message
DirList Single Click
Reset CheckPoint VPN Policy
VB6 BSTR Oddities Explained
SafeArrays in C
BSTR and Variant in C++
Property let optional args
Misc Libs
Enum Named Pipes
Vb6 Collection in C++
VB6 Overloaded Methods
EXPORT FUNCDNAME Warning
VB6 Syncronous Socket
Simple IPC
VB6 Auto Resize Form Elements
Mach3 Automation
Exit For in While
C# self register ocx
VB6 Class Method Pointers
JS Debugger
Duktape Debug Protocol
QtScript 4 VB
Vb6 Named Args
vb6 Addin Part 2
VB6 Addin vrs Toolbars
OpenFile Dialog MultiSelect
Duktape Example
DukTape JS
VB6 Unsigned
.Net version
TitleBar Height
.NET again
VB6 Self Register OCXs
Query Last 12 Mos
Progid from Interface ID
VB6 to C Array Examples
Human Readable Variant Type
ScriptBasic COM Integration
CodeView Addin
ScriptBasic - Part 2
Script Env
MSCOMCTL Win7 Error
printf override
History Combo
Disable IE
API Hooking in VB6
Addin Hook Events
FastBuild Addin
VB6 MemoryWindow
Link C Obj Files into VB6
Vb6 Standard Dlls
CStr for Pascal
Lazarus Review
asprintf for VS
VB6 GlobalMultiUse
Scintilla in VB6
Dynamic Highlight
WinVerifyTrust, CryptMsgGetParam VB6
MS GLEE Graphing
printf for VB6
C# App Config
Tero DES C# Test
VC 2008 Bit Fields
Speed trap
C# Db Class Generator
VB6 vrs .NET (again)
FireFox Whois Extension
git and vb6
Code Additions
Compiled date to string
C# ListView Sorter
VB6 Wish List
C# Process Injection
CAPTCHA Bots
C# PE Offset Calculator
VB6 Async Download
Show Desktop
coding philosophy
Code release
Dll Not Found in IDE
Advanced MSScript Control
random tip
Clipart / Vector Art
VB6 Callback from C#
Binary data from VB6 to C#
CSharp and MsScriptControl
HexDumper functions
Js Beautify From VB6 or C#
vb6 FormPos
Inline Asm w VB6
The .NET Fiasco
One rub on computers
Universal extractor