Injecting the flag (1 << 17) = 0x20000 to D3DXCreateEffect makes the shader fail compilation in another way. The only API call the game uses that I can find relies on it is D3DXGetShaderConstantTable, but the issues happen before it is ever called. Unfortunately our knowledge of DX9 is very limited, I've seen that DX9 has a flag D3DXCONSTTABLE_LARGEADDRESSAWARE but I'm not sure where exactly Its supposed to go. Forcing the engine to malloc 2GB before compilation makes the shaders fail to compile every time. The resulting pointer pointed to unallocated memory. It took a valid pointer that looked like 0x8xxxxxx3, checked that bits 0x80000003 are lit and if yes, it bit inverts the pointer and derefs it. I believe the issue is lack of Large Address Awareness: I noticed one of the d3dx9_25.dll crashes doing something that would hint as such. Usually it returns HResult "out of memory", sometimes d3dx9_25.dll prints random errors in a popup, or it just outright segfault. The issue is usually with D3DXCreateEffect. We are running into issues with an old closed-source game engine failing to compile shaders when memory nears 2GB.
0 kommentar(er)
