About our environment:
Win Server 2008 r2 64 bit, 8GB Ram
Our web server experienced a series of undetectable speed bumps for events that should have zero negative impact. One such hiccup was the update from 9.0 to 9.0.1 on March 29, 2012 which broke the CF Web Admin page. I'm not the web server admin so my ability to test for a fix was limited. Searching for a solution online was futile. The error given was:
Diagnostics | Element ESAPIUTILS is undefined in a Java object of type class [Ljava.lang.String;. <br>The error occurred on line 55. |
Eventually we requested a single incident request from Adobe tech support. The first day the support technician visited our site there was little more than just simple prodding. The technician claimed he needed to generate a VM to mimic our setup, which I still question how that was intended to be conducted. The VM creation was never fulfilled and did nothing but delay repairs due to scheduling confilicts with Adobe, our admins availability, and the weekend. Eventually the technician was able to work with us again and the CF web admin was fixed. However, before the fix was put into place the technician asked twice, once each call, if we applied any hotfixes. The answer was no both times asked on two separate days. The technician appeared to believe we applied a Hotfix due to some existence of a file somewhere in some directory that appears to me, without being said, was the indicator of his belief that a hotfix was applied which it was not.
The technician suspected the source of the problem could be:
- A hotfix applied previously, not the case.
- Permissions
- Corrupt file(s)
In the end applying a hotfix appears to be what eventually fixed our problem.
This was the processes approached during the technicians fix:
\ColdFusion9\runtime\servers\coldfusion\SERVER-INF\jrun.xml
some value was set to true to allow local web server to run.
http://helpx.adobe.com/coldfusion/kb/cumulative-hot-fix-2-coldfusion-1.html
http://helpx.adobe.com/coldfusion/kb/cumulative-hot-fix-2-coldfusion-1.html
http://helpx.adobe.com/coldfusion/kb/coldfusion-administrator-fails-permission-denied.html
Shortly after the fix over a weekend I was informed that our site was down. It appeared Coldfusion was running still and IIS had something wrong since I was getting error 503. I tested by swapping out an index.html on an unaffected site which gave back the output desired. I learned later that logs also indicated that CF had not been restarted or failed. It turned out that the IIS default application pool had stopped. Starting this again in the IIS admin console alieivated the immediate problem. However, we continued to get error 503 now and then indicating that an application pool has stopped.
DebugDiag was put into play to gather crash dumps from IIS to try and troubleshoot. When we had gathered about 5 crash dumps and each analysis had mentioned jrun_iis6_wildcard.dll in the results I suspected that something to do with the recent fix was the cause. However, when I contacted the Adobe technician on April 18th they claimed that the fix put into place could in no way affect the jrun_iis6_wildcard.dll. When I brought up that I had crash dumps from IIS with indication that the jrun connector dll was at fault he said that I would have to get in contact with Microsoft to analyze the crash dump to verify that. I'm in the process of starting a support request with Microsoft for that, but I thought this would be useful to post if anyone else has experienced this.
I have noticed since the fix also that I'm seeing error 503 in the http.log file now and then. I've also noticed that our site is far more sensative to bots visiting which I intend to address shortly to allieviate resource usage.
WinDebug analysis; tail lines of an IIS crash dump.
STACK_COMMAND: ~8s; .ecxr ; kb
SYMBOL_STACK_INDEX: 7
SYMBOL_NAME: jrun_iis6_wildcard!TerminateExtension+425
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: jrun_iis6_wildcard
IMAGE_NAME: jrun_iis6_wildcard.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4a980f56
FAILURE_BUCKET_ID: INVALID_POINTER_WRITE_c0000005_jrun_iis6_wildcard.dll!TerminateExtension
BUCKET_ID: X64_APPLICATION_FAULT_INVALID_POINTER_WRITE_jrun_iis6_wildcard!TerminateExtension+425
So my question is there any suspicion that the JRun connector is faulted or is this simply caused by excessive traffic exceeding some limits of the CF connector or do I need to focus on IIS application pool process?