Hi there,
I am running a ColdFusion server on a VPS to HostMySite.com and lately we are having strange problems with it. Every few minutes the server is not responding even if we restart the services - IIS and ColdFusion - and even the whole system.
HMS guys investigated the problem and they discovered that every connection to the server is openening multiple sockets for a single IP address (every single visitor).
Here is the full message from HMS technician, do you have any previous experience related to this?
I've been doing some advanced monitoring and troubleshooting of your VPS over the last 24 hours.
It is important to understand that the issue you're actually experiencing is related to TCP sockets. Every connection to your server opens a socket and sometimes multiple sockets for an individual IP (visitor).
I opened the sitehttp://www.viaromania.eu/and instantly there were 7 connections established from our IP address.
C:\Documents and Settings\hmsadmin>netstat -ano | find "209.41.163.23"
TCP 76.12.37.79:80 209.41.163.23:9563 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:21164 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:26819 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:36833 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:37624 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:39566 ESTABLISHED 4
TCP 76.12.37.79:3389 209.41.163.23:2577 ESTABLISHED 141388
After just browsing around a few pages on the site you can see how my connections are expanding.
C:\Documents and Settings\hmsadmin>netstat -ano | find "209.41.163.23"
TCP 76.12.37.79:80 209.41.163.23:2852 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:2900 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:11014 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:11178 TIME_WAIT 0
TCP 76.12.37.79:80 209.41.163.23:14107 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:14248 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:17177 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:17606 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:17930 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:23460 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:24594 TIME_WAIT 0
TCP 76.12.37.79:80 209.41.163.23:25191 TIME_WAIT 0
TCP 76.12.37.79:80 209.41.163.23:25507 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:32301 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:33591 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:37338 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:38404 TIME_WAIT 0
TCP 76.12.37.79:80 209.41.163.23:45140 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:49734 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:53755 ESTABLISHED 4
TCP 76.12.37.79:80 209.41.163.23:55735 TIME_WAIT 0
TCP 76.12.37.79:3389 209.41.163.23:2577 ESTABLISHED 141388
Over the last 2 days there are 205 coldfusion-out logs and they are all full of the same error:
java.net.SocketException: Software caused connection abort: socket write error
Normally when we see this we'll make a few registry adjustments that allow for more socket connections and a shorter time to live on existing socket connections. However in your case all of the registry adjustments have already been set.
MaxUserPort 65534
TcpNumConnections 200 connections
TcpTimedWaitDelay 30 seconds
I adjusted the TcpNumConnections to 500, see if this alleviates the issue. Note that allowing 500 Tcp Connections is not necessarily a good idea as this amount of traffic could theoretically bring down your server.
I created a scheduled task that executes every 60 seconds in which it counts the connections on port 80 and writes it to the file netstat.txt on the desktop.
After logging for the last 24 hours it has gone over the 500 TCP connections 19 times all between 2:21pm and 2:40pm
2:21 PM 1367
2:22 PM 1423
2:24 PM 1684
2:25 PM 1466
2:26 PM 1867so
2:27 PM 1250
2:28 PM 854
2:29 PM 796
2:30 PM 799
2:31 PM 794
2:32 PM 816
2:33 PM 730
2:34 PM 662
2:35 PM 524
2:36 PM 531
2:37 PM 539
2:38 PM 551
2:39 PM 551
2:40 PM 522
So this is pretty good news. This means your site over the last 24 hours only had 19 minutes of issues due to TCP connections.
Please, post your messages if you know why so many sockets are opened for every single IP and if this is a normal behaviour.
Greetings,
Adrian.