I’m proud to announce the avilability of our brand new forum infrastructure. We were getting really tired in the end by the lack of features of the sf.net forums, so we decided to setup FUDForum on ossim.net
I for myself am very motivated by this changes, I was getting crazy with the old environment and promised myselft to answer many more things on these new forums.
Enjoy: ossim.net forums.
Yay ! we’re proud to announce that ossim has been chosen to take part int he google summer of code program. Brian, now it’s your turn ;-).
I’ll post another entry when we’ve got more information about how this works.
Congratulations! Your organization "OSSIM: Open Source Security Information Management" has been accepted in to the Google Summer of Code(tm) 2008. You have been assigned as primary point of contact and as an administrator for your organization. please visit http://code.google.com/soc/mentor_step1.html and sign up using your Google Account. Thanks. - Your friendly Google Summer of Code administrators
Remember the couple of posts I made back in November in the tuning section ? Well, I finally got the time to look into this issue again and have made some interesting discoveries the last couple of days. I’m really enjoying this.
The following table illustrates some comparisons between a stock Base 1.3.9 (ossim patched) and the tuned rewrite I’ve got running right now. These optimizations are now part of our appliance offering (updates for already deployed ones on the way) and will be released to the public afterwards. Thanks to everybody that has been helping me on this, specially to the people at #ossim in freenode ;-).
BREAK
Just for statistical issues, the current contents of the database are (links are broken of course):
| Sensors/Total: 470 / 1692 Unique Alerts:
|
Traffic Profile by Protocol
|
Here are the results of some tests I’ve made, you’ll have to trust my word on these :-). The testing host is a Pentium M 2.0Ghz with 1GB RAM, with query caching disabled. If you want me to post some specific test or logs as proof of concept please don’t hesitate to ask.
| Loading Times (in seconds) | ||
|---|---|---|
| ACID/BASE 1.3.9 (ossim patched) | AV Tuned ACID/BASE | |
| Front Page | 185 | 1 |
| Unique Alerts | 44 | 4 |
| Source IPs | 32 | 3 |
| Source IPs (order by count(dest)) | 76 | 6 |
| Dest Ports (order by ocurrences) | 105 | 6 |
| Random Event Detail | 42 | 1 |
| Next Event Detail | 23 | 1 |
Those numbers are with a couple of days worth of data on a Mysql 5.0.32, not taking other optimizations into account which vastly improve that performance on the applliances such as:
Well, just wanted to share those numbers since forensic database performance has been a real pita for many years and I’m really happy it’s starting to work right now on large amounts of data.
Happy spring break to everyone. Aaah, and check the GSOC page on monday, hopefully we’re eligible as a mentoring organization for this years summer of code, check out our ideas here.
A couple of days ago I was fixing the fortinet/fortigate with the kind help of a Swiss OSSIM user (thanks Mikael ;-) ) and I wrote this little piece of python in order to help me out with it. Now I’m using it a lot to debug plugins so I guess more people could benefit from this also :-)
And well, I’ll paste a sample plugin debugging session in order to give ideas.
BTW, this assumes basic knowledge of regular expressions, check this Regexp Primer out if you want to refresh that knowledge. And BTW2, some log lines are broken for readability.
BREAK
#!/usr/bin/python
import sys,re
if sys.argv[3] is None:
print "Args are filename, regexp and [0|1]"
f = open(sys.argv[1], 'r')
data = f.readlines()
exp=sys.argv[2]
print sys.argv[2]
line_match = 0
matched = 0
for line in data:
result = re.findall(exp,line)
try:
tmp = result[0]
except IndexError:
if sys.argv[3] is "1":
print "Not matched:", line
continue
print result
matched += 1
print "Counted", len(data), "lines."
print "Matched", matched, "lines."
Basically it will take a logfile as input, a regexp and wether to verbosely show the matched lines or not. That way you can start working towards more complex regexps and test it against a full logfile in realtime.
regexp.py logfile “regexp” (0|1) # 0 == do not show “non-matching lines”
Let’s go through a simple file containing logs (got it from http://www.ossec.net/wiki/index.php/PostgreSQL_Samples).
[2007-08-31 19:22:21.469 ADT] :[unknown] LOG: connection received: host=192.168.2.99 port=52136
[2007-08-31 19:22:21.485 ADT] 192.168.2.99:ossecdb LOG: connection authorized: user=ossec_user
database=ossecdb
[2007-08-31 19:22:22.427 ADT] 192.168.2.99:ossecdb LOG: disconnection: session time: 0:00:00.95
user=ossec_user database=ossecdb host=192.168.2.99 port=52136
[2007-09-27 11:02:44.941 ADT] 192.168.2.10:ossecdb ERROR: relation "lala" does not exist
[2007-09-27 11:02:46.444 ADT] 192.168.2.10:ossecdb LOG: disconnection: session time: 0:00:35.79
user=ossec_user database=ossecdb host=192.168.2.10 port=3584
Log messages:
[2007-09-01 07:14:41.062 ADT] : LOG: autovacuum: processing database "template1"
[2007-09-01 07:15:41.079 ADT] : LOG: autovacuum: processing database "ossecdb"
Query log:
[2007-09-01 16:44:49.244 ADT] 192.168.2.10:ossecdb LOG: duration: 4.550 ms statement:
SELECT id FROM location WHERE name = 'enigma->/var/log/messages' AND server_id = '1'
[2007-09-01 16:44:49.251 ADT] 192.168.2.10:ossecdb LOG: duration: 5.252 ms statement:
INSERT INTO location(server_id, name) VALUES ('1', 'enigma->/var/log/messages')
[2007-09-01 16:44:49.252 ADT] 192.168.2.10:ossecdb LOG: duration: 0.016 ms statement:
SELECT id FROM location WHERE name = 'enigma->/var/log/messages' AND server_id = '1'
[2007-09-27 11:02:51.611 ADT] 192.168.2.10:ossecdb LOG: statement: INSERT INTO
alert(id,server_id,rule_id,timestamp,location_id,src_ip)
VALUES ('3577', '1', '50503','1190916566', '140', '0')
Query error:
[2007-08-31 19:17:42.128 ADT] 192.168.2.99:test ERROR: relation "alertaaa" does not exist
[2007-08-31 19:17:46.375 ADT] 192.168.2.99:test ERROR: syntax error at or near "a" at character 1
[2007-09-27 11:02:44.941 ADT] 192.168.2.10:ossecdb ERROR: relation "lala" does not exist
Authentication error:
[2007-09-01 19:08:49.862 ADT] : LOG: connection received: host=192.168.2.99 port=37142
[2007-09-01 19:08:49.869 ADT] 192.168.2.99: FATAL: password authentication failed for user "ossec_user"
Gestalt:tmp dk$ regexp.py postgresql.log.txt "(^.*$)" 0 | tail -n 2 ['[2007-09-01 19:08:49.869 ADT] 192.168.2.99: FATAL: password authentication failed for user "ossec_user"', '', ''] Counted 25 lines. Matched 25 lines. Gestalt:tmp dk$
The tail after the command is there because by default it shows all the matching lines, and one is enough for our sample.
The used regexp is (^.*$) which as you know matches everything from the beginning to the end.
Gestalt:tmp dk$ regexp.py postgresql.log.txt "^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+(.*)$" 0
^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+(.*)$
[('2007-08-31 19:22:21', ':[unknown] LOG: connection received: host=192.168.2.99 port=52136')]
[('2007-08-31 19:22:21', '192.168.2.99:ossecdb LOG: connection authorized:
user=ossec_user database=ossecdb')]
[('2007-08-31 19:22:22', '192.168.2.99:ossecdb LOG: disconnection: session time: 0:00:00.95
user=ossec_user database=ossecdb host=192.168.2.99 port=52136')]
[('2007-09-27 11:02:44', '192.168.2.10:ossecdb ERROR: relation "lala" does not exist')]
[('2007-09-27 11:02:46', '192.168.2.10:ossecdb LOG: disconnection: session time: 0:00:35.79
user=ossec_user database=ossecdb host=192.168.2.10 port=3584')]
[('2007-09-01 07:14:41', ': LOG: autovacuum: processing database "template1"')]
[('2007-09-01 07:15:41', ': LOG: autovacuum: processing database "ossecdb"')]
[('2007-09-01 16:44:49', "192.168.2.10:ossecdb LOG: duration: 4.550 ms statement:
SELECT id FROM location WHERE name = 'enigma->/var/log/messages' AND server_id = '1'")]
[('2007-09-01 16:44:49', "192.168.2.10:ossecdb LOG: duration: 5.252 ms statement:
INSERT INTO location(server_id, name) VALUES ('1', 'enigma->/var/log/messages')")]
[('2007-09-01 16:44:49', "192.168.2.10:ossecdb LOG: duration: 0.016 ms statement:
SELECT id FROM location WHERE name = 'enigma->/var/log/messages' AND server_id = '1'")]
[('2007-09-27 11:02:51', "192.168.2.10:ossecdb LOG: statement:
INSERT INTO alert(id,server_id,rule_id,timestamp,location_id,src_ip)
VALUES ('3577', '1', '50503','1190916566', '140', '0')")]
[('2007-08-31 19:17:42', '192.168.2.99:test ERROR: relation "alertaaa" does not exist')]
[('2007-08-31 19:17:46', '192.168.2.99:test ERROR: syntax error at or near "a" at character 1')]
[('2007-09-27 11:02:44', '192.168.2.10:ossecdb ERROR: relation "lala" does not exist')]
[('2007-09-01 19:08:49', ': LOG: connection received: host=192.168.2.99 port=37142')]
[('2007-09-01 19:08:49', '192.168.2.99: FATAL: password authentication failed for user "ossec_user"')]
Counted 25 lines.
Matched 16 lines.
Gestalt:tmp dk$
Here we make a first attempt at separating the original datetime from the rest of the line. Additionally we’ve filtered out the junk lines that might appear in the file, so now we only match 16 lines (the actual 16 lines that contain valid log files). Out regexp starts looking uglier now, but still understandable ;-): ^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+(.*)$
Gestalt:tmp dk$ regexp.py postgresql.log.txt
"^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+(?P<dbhost>[^:]+)?:(?P<dbname>\S+)?\s+(.*)$" 0 | tail -n 3
[('2007-09-01 19:08:49', '192.168.2.99', '', 'FATAL: password authentication failed for user "ossec_user"')]
Counted 25 lines.
Matched 16 lines.
This time I’m facing the first problems. In order to get the output shown above I had to go through a bit of try and error.
Starting with a first try I see there are 5 lines not matching, since we know it’s 16 we need to match:
Gestalt:tmp dk$ regexp.py postgresql.log.txt "^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+([^:]+):(\S+)\s+(.*)$" 0 | tail -n 2 Counted 25 lines. Matched 11 lines. Gestalt:tmp dk$
Let’s see which ones are not matching (notice the change from 0 to 1 at the end of the python’s ARGV):
Gestalt:tmp dk$ regexp.py postgresql.log.txt "^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+([^:]+):(\S+)\s+(.*)$" 1 | grep "Not matched: \[" Not matched: [2007-08-31 19:22:21.469 ADT] :[unknown] LOG: connection received: host=192.168.2.99 port=52136 Not matched: [2007-09-01 07:14:41.062 ADT] : LOG: autovacuum: processing database "template1" Not matched: [2007-09-01 07:15:41.079 ADT] : LOG: autovacuum: processing database "ossecdb" Not matched: [2007-09-01 19:08:49.862 ADT] : LOG: connection received: host=192.168.2.99 port=37142 Not matched: [2007-09-01 19:08:49.869 ADT] 192.168.2.99: FATAL: password authentication failed for user "ossec_user" Gestalt:tmp dk$
Ok, understood, either DB host or DB name are optional, so let’s add that to our regexp, getting a final regexp of: ^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+(?P<dbhost>[^:]+)?:(?P<dbname>\S+)?\s+(.*)$
On the matching log line we can also see how the db host and name are being correctly extracted ;-).
Gestalt:tmp dk$ regexp.py postgresql.log.txt
"^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+(?P<dbhost>[^:]+)?:(?P<dbname>\S+)?\s+(?P<sid>[^:]+):\s+(?P<log_msg>.*)$" 0 | tail -n 3
[('2007-09-01 19:08:49', '192.168.2.99', '', 'FATAL', 'password authentication failed for user "ossec_user"')]
Counted 25 lines.
Matched 16 lines.
Gestalt:tmp dk$
Looking at the remainder string it starts getting obvious that the next field is sort of a priority value. In this first phase the postfix plugin will remain somewhat crippled since I’m going to use that field as plugin sid, instead of identifying each of the log types by what they actually do. (That’s left as a task for the reader).
Our resulting regexp, extracting the priority and removing some white space in front of the logline would be: ^\[(?P<date>\S+\s+\S+)\.\d+\s+ADT\]\s+(?P<dbhost>[^:]+)?:(?P<dbname>\S+)?\s+(?P<sid>[^:]+):\s+(?P<log_msg>.*)$
The plugin is far from being finished, but the goal of this tutorial was to demistify the regexp part a bit, since it’s actually only based on a bit of research about the log (possible values for fields), a bit of intuition and lots of try && error.
I hope that goal has been met.
To be continued… (finishing the regexp, writing the .cfg file and writing the .sql file)
I wanted to point you at two things I think that are important, things that we’ve been neglecting in the past months.
So, if you’ve got bugs or suggestions for ossim, please start posting them there. And if you just want to hang out with us join on the IRC nchannel.
During the next days I’ll post an update on currenct development, we’re working on some exciting features right now ;-)
