Shellcode Hook Detection
Author: David Zimmer
Date: 05.15.12 - 7:43pm
Every now and then shellcode pops up that does hook detection.
Ok fine, scdbg doesnt use jmp hooks..but not so fine if the shellcode doesnt actually check for a hook, and just always replaces the api prolog and does a jmp api+5. This screws up scdbg because the hook address isnt hit and dll memory starts getting executed.
I pushed a test patch tonight to the github repository that aims to fix this. If it detects any api+5 address being executed, it assumes its a hook detection code running, and will pass execution back to a hookDetect call back which tries to restore the stack and allow the standard api logging code to run normally.
Tested it with several shellcodes and seems to be working fine.
A precompiled binary is available in the github archive you can download if you want to play with it. Of the thoughts I have had on how to handle this, this seems to be the best solution so far...
These types of hook evasion routines will also cause a crash in sclog. Initially i switched to a 7 byte push const ret hook so I wouldnt trigger the hook check, but if they jmp +5 regardless its a crash. For sclog, I guess what I should do is pad the beginning of the hook a bit so that jmp +5 lands them squarely on my jmp hookthunk. That would be the best of both worlds.
Update: I tried the 7 byte hook, it worked, but not all api have enough room to embed it. Its really best to have a prolog pad that is devoid of instructions like jumps and calls to slip in a hook. I could have it selectivly fall back to original style hooks, but in the end i just reverted to standard detours style hooks for simplicity and stability.
One additional tip, if you get a shellcode which starts executing dll opcodes and then crashs in scdbg. You can find the point of transfer to dll memory quite quickly with the following procedure:
Additionally if the problem is when executing null memory, there is a special switch to break on that, replace step 1 with the -b0 option.
Couple examples for reference:
seg000:0000019F HookCheck proc near seg000:0000019F seg000:0000019F cmp byte ptr [eax], 0E8h ; F seg000:000001A2 jz short loc_1AE seg000:000001A4 cmp byte ptr [eax], 0E9h ; T seg000:000001A7 jz short loc_1AE seg000:000001A9 cmp byte ptr [eax], 0EBh ; d seg000:000001AC jnz short loc_1BF seg000:000001AE seg000:000001AE loc_1AE: seg000:000001AE seg000:000001AE cmp dword ptr [eax+5], 90909090h seg000:000001B5 jz short loc_1BF seg000:000001B7 mov edi, edi seg000:000001B9 push ebp seg000:000001BA mov ebp, esp seg000:000001BC lea eax, [eax+5] seg000:000001BF seg000:000001BF loc_1BF: seg000:000001BF seg000:000001BF jmp eax seg000:000001BF HookCheck endp