AlienVault R&D Labs Portal. Get the latest news from our research.
Header

Tutorial 8: OSSIM + JAsperServer + iReport Tutorial

March 17th, 2009 | Posted by DK in Tutorials - (2 Comments)

This eigth installment of the tutorial series will focus on a feature which will be revolutionary for OSSIM for sure: tight jasperserver integration for custom/periodic reports with the guarantee of a strong BI suite. The upcoming installer release will include both Tomcat as well as JasperServer ready to use and with sample preloaded reports and datasources. (Note: Installer beta2 users can already test some of this out, although no real tight integration until beta4 will be in place).

If you haven’t heard about JasperServer nor iReport you can them check out JasperServer and iReport for some background. Quoting those two pages:


“JasperServer is a high-performance business intelligence platform and report server designed for developers and businesses. Deploy JasperServer when end-users need to create their own ad hoc queries, reports, charts, crosstabs, dashboards, or it becomes necessary to secure, store, schedule, distribute, share, drill-down, or interact with reports.”

“iReport is a graphical report tool for report designers, developers, and power-users. iReport provides complete coverage of all the reporting capabilities in JasperReports, JasperServer, and Jasper4Salesforce, including the creation of parameterized reports, pixel-perfect production reports, and remote JasperServer repository management”.

I’m no jasperreports expert myself, I used it for report creation and I’m sure there are tons of tips and tricks experts can provide. Any comments and feedback that help improving this article will be greatly appreciated :-)

During this tutorial the following steps will be covered:

  • OSSIM – iReport – JasperServer integration
  • Assumptions
  • iReport download and setup
  • OSSIM for iReport setup
  • Sample report: top events by risk/ocurrence.
  • Uploading this report to jasperserver
  • What to do next?
  • Tips and Tricks.
  • Sharing your work/fun: the Alien Forge.

BREAK

OSSIM – JasperServer integration

Ok, a couple of notes first. The installer uses JDK6, Tomcat 6.0.18 and JasperServer 3.1. Everything is installed into /var/tomcat/, passwords get adjusted from /etc/ossim/ossim_setup.conf using ossim-reconfig and default login/password is jasperadmin/jasperadmin. Point your browser at http://ossim_ip:8080/jasperserver/ and you’re set.

OSSIM will provide a series of default datasources for the most useful databases for reporting, that is, ossim, ocs inventory and snort.

The idea behind this provide strong reporting without having to code it from zero up, but also have a way to share generic reports with other users. That’s what we’re going to do using the .jrxml formats, be able to share reports with others, creating an amazing amount of custom reports for users to use. More on this on the last section, the Alien Forge ;-)

Warning: the following applies to users of the installer beta versions released on 2009/03 and beyond, don’t try this with 1.0.6 or previous versions; you’ll end up quite frustrated.

Assumptions

  • My IP is 192.168.1.24
  • OSSIM is at 192.168.1.22
  • User is ‘root’
  • Password is ‘password’
  • Jasperserver credentials are ‘jasperadmin:jasperadmin’

iReport download and setup

First, get iReport for your platform: Download. Win and mac got an easy installer, for linux I guess it’s the .jar file anyway. (Haven’t tested it).

OSSIM for iReport setup

In order to enable access from your client you’ve got to grant it permissions from within mysql and enable mysql to listen on the network.

  1. Log into OSSIM via ssh as root.
  2. Edit /etc/mysql/my.cnf, search the line bind-address … and replace 127.0.0.1 with 0.0.0.0. Execute /etc/init.d/mysql restart. This will enable remote connections to your mysql, so please choose a different password for the remote user in case somebody connects using your ip.
  3. Execute “ossim-db” (this will read the password from /etc/ossim/ossim_setup.conf and drop you into mysql>)
  4. Grant access to your client host. You might need to add another user here for our tutorial if you’re reading this before the final release, more on this later: “grant all on *.* to root@192.168.1.24 identified by ‘password’;“

Note: this is the easiest way of doing it, tighter permissions and separate users for each DB are recommended in high security environments

Next, fire up iReport. We’ll start by creating a blank report:
(Image removed, broken link, I’m very sorry. DK.)

Let’s call it “Top20EventReport” for example (although later it will be top 10 and 50, but I had the screenshots done already so at least on my side it will be called like that :P).

Here goes our next screen:
(Image removed, broken link, I’m very sorry. DK.)

Next we’ll add both datasources to our iReport. Click on the “report datasources button” as seen here:

(Image removed, broken link, I’m very sorry. DK.)  and fill in the OSSIMDS definition with your info:
(Image removed, broken link, I’m very sorry. DK.).

Rinse and repeat for snort and you should end up with the following:
(Image removed, broken link, I’m very sorry. DK.).

Sample report: top events by risk/ocurrence

Our report will have two sections:
- Top 50 events by risk (from acid_event).
- Top 10 events by occurrence.

First we start off with the query(s). This might be the hardest part of all, but as said, the idea is that all ossim users share their reports, so there might be those with more SQL knowledge, those with report designing skills and those with a greater security knowledge; combining the three of them will yield great reports.

Our main report will be the one just reporting the 50 events. After that we’ll include a sub-report with the events by occurrence.

For the record, the table we’re going to use herein is:

acid_event:
----------------------------------------
| Field | Type | Null | Key | Default | Extra |
+-------------------+------------------+------+-----+---------+-------+
| sid | int(10) unsigned | NO | PRI | NULL | |
| cid | int(10) unsigned | NO | PRI | NULL | |
| signature | int(10) unsigned | NO | MUL | NULL | |
| sig_name | varchar(255) | YES | MUL | NULL | |
| sig_class_id | int(10) unsigned | YES | MUL | NULL | |
| sig_priority | int(10) unsigned | YES | MUL | NULL | |
| timestamp | datetime | NO | MUL | NULL | |
| ip_src | int(10) unsigned | YES | MUL | NULL | |
| ip_dst | int(10) unsigned | YES | MUL | NULL | |
| ip_proto | int(11) | YES | MUL | NULL | |
| layer4_sport | int(10) unsigned | YES | MUL | NULL | |
| layer4_dport | int(10) unsigned | YES | MUL | NULL | |
| ossim_type | int(11) | YES | MUL | 1 | |
| ossim_priority | int(11) | YES | MUL | 1 | |
| ossim_reliability | int(11) | YES | MUL | 1 | |
| ossim_asset_src | int(11) | YES | MUL | 1 | |
| ossim_asset_dst | int(11) | YES | MUL | 1 | |
| ossim_risk_c | int(11) | YES | MUL | 1 | |
| ossim_risk_a | int(11) | YES | MUL | 1 | |
------------------------------------------

Everything we need for this report can be found in there.

Our querys would be as follows:

select timestamp, sig_name, inet_ntoa(ip_src) as source, layer4_sport, inet_ntoa(ip_dst) as dest, layer4_dport,
((ossim_risk_c + ossim_risk_a) / 2) as risk from acid_event where timestamp > DATE_SUB(NOW(), INTERVAL 7 DAY)
order by risk desc limit 50;

select sig_name, count(*) as cnt from acid_event where timestamp > DATE_SUB(NOW(), INTERVAL 7 DAY) group by sig_name
order by cnt desc limit 10;

So, first let’s select SNORTDB as datasource on top. Then right-click on report name, “Edit Query” and insert our first one and click OK. Edit query location can be found here:
(Image removed, broken link, I’m very sorry. DK.)

Our first test would be to add some fields to the detail. Click on “Fields” so that you get a list like this:

  • timestamp
  • sig_name
  • source
  • layer4_sport
  • dest
  • layer4_dport
  • risk

Or, on image:
(Image removed, broken link, I’m very sorry. DK.)

If we move any of those into the “Detail” (or any other) band, we’ll get them into our report. Let’s move timestamp, sig_name and risk into it, making the sections shorter and see how it will look. Also, a quick title (using a “Static text” field from the palette into title) and removing unused sections will help. Click on Preview next :-)
(Screens below show the final layout and how our first preview should look like:)


(Image removed, broken link, I’m very sorry. DK.)

There we go ! Some first (ugly :P) results.

Next we’ll get the top 10 graph into the report to get some colour into it.

Let’s start by generating a sub-report.
Get back to designer view, make the Title bar bigger and drag a sub-report item from palette into it:

(Image removed, broken link, I’m very sorry. DK.)

Start of with selecting “Create a new report”. Next we’ll select SNORTDB as datasource, since acid_event is inside SNORTDB:

(Image removed, broken link, I’m very sorry. DK.).

Then paste our sub-query into it:

select sig_name, count(*) as cnt from acid_event where timestamp > DATE_SUB(NOW(), INTERVAL 7 DAY)
group by sig_name order by cnt desc limit 10;

We’ll need both fields, so move them both to the right
(Image removed, broken link, I’m very sorry. DK.).

No need for grouping, same connection as master and we’re set. Next, we’ll clean up everything, just leaving the title band. Select everything, right click, delete:

(Image removed, broken link, I’m very sorry. DK.)

A nice empty drawing space:
(Image removed, broken link, I’m very sorry. DK.). Let’s fill it.

See the chart widget on the right area ? Drag it right into our blank area and you’ll see a popup with the selection.

We’ll want a 3D pie first:
(Image removed, broken link, I’m very sorry. DK.).

Next we’ll set up some common properties. Feel free to try out variations of course :-).
Click on our chart and notice that the properties window on the right changes to “3D Pie Chart – Properties”. We’ll touch some of them in there.

  • Find show legend, enable it.
  • Find Theme and change it to eye.candy.sixties
  • Find legend position, switch it to right.

Next thing to do is to stretch the chart to fit into your window. Result should be something like this:

(Image removed, broken link, I’m very sorry. DK.)

Let’s fill in the values next. Right click chart, click on “edit chart data” and then on the “Detail tab”. Remember we had two fields available to us on this query, sig_name and cnt ? Well, let’s fill them into the right place.

The key on which we’ll iterate is clearly the actual name, while the value will be the count. Since we also want to see how many there are of each count, we’ll use the count as label expression too. Note: label has to be a string so we need to convert the field into the right value before using it. You can see the final positions below:

(Image removed, broken link, I’m very sorry. DK.) Accept and preview this new graph. It should ressemble this one:
(Image removed, broken link, I’m very sorry. DK.)

There we go, our graph is ready. Numbers are a bit crippled but I don’t know how to fix this right now, it’s a start nonetheless and I’ll update when I figure out (or hopefully someone else figures it out and tells me…).

Now let’s get back to the main report (look at the tabs on top). Adjust the graph to content and align it to the left of our doc.

(Image removed, broken link, I’m very sorry. DK.)

Just a quick preview and… voilá, (hopefully) you’re seeing your shiny new report.
(Image removed, broken link, I’m very sorry. DK.)

Uploading this report to jasperserver

Now we want to get this onto our server, see that we can generate it there whenever we want and schedule reports so they get sent out to the manager(s) every once in a while. (Note: don’t send them out this way, add some corporate logos, adjust styles, include more info and all that of course :-))

So, we start selecting the item “JasperServer Repository” from the “Window” menu:
(Image removed, broken link, I’m very sorry. DK.).

This will open up a new tab on the left side of the screen.
There we’ll click on “Add new server”
so we can connect to our running jasperserver and fill in our jasperserver configuration:

(Image removed, broken link, I’m very sorry. DK.)

adding the right ip, removing the “-pro” behind jasperserver and entering credentials (jasperadmin:jasperadmin by default):

(Image removed, broken link, I’m very sorry. DK.)

Next comes a fork in the tutorial. If you click on your server and got OSSIMDS and SNORTDS as datasources, then your installed version is later than beta2 and you can skip below to the “Add report unit” part. Otherwise you’ll need to define the two datasources.

First get back into ossim at console, enter “ossim-db” and create a user for local reporting access (as said before, it’s recommended to limit the permissions on production environments, but this is easier to debug):

grant all on *.* to reports@127.0.0.1 identified by 'report_password';
grant all on *.* to reports@localhost identified by 'report_password';

After this, in order to create the missing datasources, we’ll right click on our server and Add new Datasource

(Image removed, broken link, I’m very sorry. DK.)
.

Fill in the name and label (OSSIMDS) and get into detail. We can import it from our report first but need to change the ip to 127.0.0.1.

Rinse and repeat for SNORTDS. (Image removed, broken link, I’m very sorry. DK.)

If you click on your server, you should now have two datasources (I’ve got no idea how the “Add new server” label got on top of OSSIMDS, but trust me, it’s there) defined.

So now we can add a report unit (the combined report we just created) by right clicking again and selecting it.

(Image removed, broken link, I’m very sorry. DK.).

We’ll give it a name (names can’t contain spaces I believe), a label, a description and continue.

(Image removed, broken link, I’m very sorry. DK.).

Next we’ll select “locally defined” and load our main report .jrxml file into it (not the subreport one).
(Image removed, broken link, I’m very sorry. DK.).

After that we’ll get asked for the datasource. Since we’re reading from snort, we’ll select SNORTDS from the repository.
(Image removed, broken link, I’m very sorry. DK.).

Go ahead and it will prompt about a missing subreport and wether it shall be included too. Of course it should, so just accept:
(Image removed, broken link, I’m very sorry. DK.).

There we go, our report ready to be used along with two datasources.
(Image removed, broken link, I’m very sorry. DK.)

Let’s see if it worked. Point your browser at http://your_ip:8080/jasperserver/ and log in using jasperadmin/jasperadmin. As you (hopefully will) see, there is our report. Let’s generate it by clicking on the name :-)

(Image removed, broken link, I’m very sorry. DK.)

Aaah, there it is
(Image removed, broken link, I’m very sorry. DK.)
. And in flash, which is prettier:
(Image removed, broken link, I’m very sorry. DK.)

Lastly I’d suggest getting back to iReport and upload the .jrxml reports files as backup, so next time you can fetch them from there and you can share it with co-workers:

(Image removed, broken link, I’m very sorry. DK.)

What to do next?

Well, next steps would be scheduling reports to run on a daily/weekly/monthly basis as well as getting those reports into the executive panel for real-time visualization.

Tips and Tricks

The next list is intended to be a quick lifesaver guide for those into jasperserver. I’m sure I’ll need it in 6 months :-)

  • Jasper is at /var/tomcat/. (easy one :P)
  • Main logs are at /var/tomcat/logs/catalina.out.
  • DB passwords get stored at three places: /var/tomcat/conf/Catalina/localhost/jasper… (verify) (add other two).
  • If in trouble, get the latest jasperserver .war, undeploy, deploy and reconfig.
  • Don’t try to move elements among document spaces (detail, title, group header, etc…), it will fail.
  • Create subreports if unsure, they’re easy to maintain.
  • If you want to include a report into executive panel, schedule it to overwrite itself, grab the url and add “j_username and j_password” to it’s url, as in < img src=”http://192.168.1.22:8080/jasperserver/fileview/fileview/Reports/Report1.html_files/img_0_0_0?j_username=jasperadmin&j_password=jasperadmin” >
  • If you’re generating a report for management, use as many colours as possible; they love it!
  • No need to save, every preview Saves. (I hope nobody gets offended, but this phrase just reminded me too much of Jesus saves… after each level he passes.

Sharing your work/fun: The Alien Forge

Why do I call this work/fun? My reasoning behind this whole article/post is as following: I’ll do a sample report and share it with all the fellow ossim users. This report might look like shit and be of a limited use, but I hope to get something moving. This report just shows a weekly top 50 events by risk and a top 10 events by raw event count. The design is horrible and the usefulness is limited, but what if…

  1. somebody took this one, added some nice column layout, some colors, tweaked a couple of things and uploaded it to the forge again ?
  2. somebody took the next one, added a listing for top 10 hosts for each of those events to the report including a name resolve?
  3. someone else took the previous one and added a field that would prompt the user for a month and it would generate a report for that one?
  4. take 10 more iterations…
  5. somebody integrates 10 of these sub-reports and creates an awesome auto-report for everyone to schedule on a daily/weekly/montly basis?

This last step is my goal and I really think it can be done.

OSSIM parts that benefit from custom content:

  • Snort
  • Nessus
  • Ossim directives
  • Ossim plugins (cfg + sql)
  • Cross correlation rules
  • Inventory Correlation rules
  • Event repriorizations
  • OSSEC plugins
  • JasperServer custom reports

The needs for this custom content will be covered in two ways:

  • AlienVault Feed
  • Alien Forge

The AlienVault Feed will include updates for all those items on a regular basis. Updates will happen once a week with exceptional “critical” updates on certain important threats. Most of the items from the AlienVault Feed will be consolidated into the ossim releases, effectively creating a delay for regular users.

The Alien Forge on the other hand is an exchange place for all those items between users. Got something to share? put it in there. The more content is shared the more people will be willing to share back. Think of a snowball rolling down a snowfield :-)

The code is just being finished. Uploads will be categorized into Manufacturer, Product, Version and Type. Everybody will get full credit for their uploads and points, points we plan on being able to spend in:

  • Free subscriptions.
  • Certification extensions.
  • Shirts / toys.
  • Free Appliances.
  • A free tryp (business class) to Mars once Mars flights are of common use (a visit to AlienVault headquarters included :P)

Following are a couple of screens of the current status:

I hope you liked this tutorial, the best way you can show that is by contributing reports back, any of them, even if it’s a “last 10 entries into the mysql users table”; everything will be used at some time by another users :-) Thanks for reading this far btw, it’s been one of the longest and most intensive tutorials so far.

DK

Mr Wolf Wannabe.

More Posts - Website

, ,

Tutorial 7: Feature highlight / pre-tutorial on Risk Maps

October 15th, 2008 | Posted by DK in Tutorials - (Comments Off)

Introduction

Today I would to share something interesting we’re working on: Risk/Availability/Vulnerability indicator Maps.

The purpose was to fit the most important information that can be gained from ossim all over it’s interface, into a simple to use, simple to manage and simple to analyze interface.

We already had an approach to both, to using maps (images) and to aggregate/organize different input into meta-objets (what we called business processes). But, both of them had the same problem: they were complex and they were ugly.

BREAK

So now, using the data from that part, we tried to make an interface that was as appealing to the user as possible, but which also was foolproof so that it could be used by less-skilled users (*cough* management *cough*). And here is the result.

I’d love to express my sincere gratitude to Juan Manuel Albarracin, who’s coded up the groundworks for all of this in less than four days. Kudos :-).

Also, the screenshots shown here might not reflect the final release. This is work in progress, I’m going to commit code for it to the cvs very soon (tomorrow or monday/tuesday) and it will be in the 1.1 installer release, but of course we”ll be polishing the look and functionality before that.

Sample setup

Final result

The screenshot below these lines shows a finished map of our office, with icons matching people and some specific hosts/environments.

(Image removed, broken link, I’m very sorry. DK.).

On that map you can see the basic object which make up the new map:

  1. The background shape. Usually this would be a network map for your infrastructure, but can be anything from geopolitical maps, to logos, to blank pages or whatever you want.
  2. Configurable icons. A standard set will be provided, custom icons can be uploaded.
  3. Each element can be freely dragged around (during configuration)
  4. Every element can be linked to some url (some part of ossim, another map, an external place)
  5. Each element features it’s own easy to understand (green/yellow/red :P) Risk/Vulnerability/Availability indicators

Configuration

Our next screenshot features the configuration interface for all of this.

(Image removed, broken link, I’m very sorry. DK.) And again, with the upload part collapsed:
(Image removed, broken link, I’m very sorry. DK.).

You’ve got options to upload maps and icons (icons require a custom name). After having uploaded all the maps you want to configure and all the icons you want to use (besides the default provided ones), you’re ready to go.

Toggling away the maps section (we’ll replace that nasty link with something niftier) would be a nice first step, leaving only the map and the lower section.

After this, we’ve got four things to decide:

  1. Choose an icon
  2. Choose an element to report on (more on this later)
  3. Assign a name
  4. Link to another map or to a random url (want to exted this to provide links to standard places across ossim)

Except the link url, the other three elements are mandatory.

The next image shows the nifty icon selector, which uses Lytebox as backend. It allows you to choose icons of your own or select from a set of default icons.

(Image removed, broken link, I’m very sorry. DK.).

As for objects, those usable for placement include:

  • Hosts
  • Networks
  • Host groups
  • Network groups
  • Servers
  • Sensors
  • Businessprocesses

Once that is choosen, you’ll add a new indicator. This will place the icon along with it’s indicator on the map, allowing you to move it

Last but not least, there’s also the ability to add “hot-zones” for those areas where you want to delve into deeper detail. The next image shows a square around Australia, which we could link to a site with info about it, or even better, to a more detailed map with provices/areas and indicators.

(Image removed, broken link, I’m very sorry. DK.).

This has been a brief introduction to the subject, more in-depth information along with how those three final indicators are being calculated will be posted on a follow-up.

From here on what you do is up to your imagination, I guess more obvious environments are those mapping a high level network map of your company to some lower level ones, drilling down on host and network status that way.

DK

Mr Wolf Wannabe.

More Posts - Website

, , ,

Tutorial 4: Correlation engine primer

December 18th, 2007 | Posted by DK in News - (Comments Off)

Introduction

In order to answer to a recent forum post I had to do a quick research since it had been some time since I last tested this.

The exact question was:

Hello,

Is there a document talking about how the directives are processed?  One question
that I have is if you have multiple directives created and an event comes in
that matches the initial states of more than a single directive will both actually
process the event, or only the first match (which I think is the case)?

Thanks for any clarification you can provide.

Stephen

This post gives a bit of insight to how the correlation engine works and features some simple, custom made directives that help me answer that question.

The test environment features two events belonging to the ssh plugin (plugin_id 4003):

  • SSH password failed (plugin_sid 1)
  • SSH password accepted (plugin_sid 7)

In order to test this I’ve created three directives (plugin_id 1505)

  • Test directive 21, grouping one login failure and one success
  • Test directive 22, grouping one login failure and one success
  • Test directive 23, used in the second case, grouping those two

So, with all of this in place it was easy to simulate this failing a login and succeeding afterwards.

BREAK

Case 1: Two concurrent correlation rules

Directives

Put these into generic.xml for example 

&lt;directive id="21" name="Testing concurrency 1" priority="5"&gt;
  &lt;rule type="detector" name="Level 1:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" &gt;
        &lt;rules&gt;
                &lt;rule type="detector" name="Level 1:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" /&gt;
        &lt;/rules&gt;
  &lt;/rule&gt;
&lt;/directive&gt;

&lt;directive id="22" name="Testing concurrency 2" priority="5"&gt;
  &lt;rule type="detector" name="Level 2:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" &gt;
        &lt;rules&gt;
                &lt;rule type="detector" name="Level 2:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" /&gt;
        &lt;/rules&gt;
  &lt;/rule&gt;
&lt;/directive&gt;

Logs

2007-12-10 04:19:56 OSSIM-Message: Event received: event id="0" alarm="0" type="detector"
date="2007-12-10 04:19:53" plugin_id="4003" plugin_sid="1" src_ip="10.0.1.3" src_port="64270"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" asset_src="1" asset_dst="1"
log="Dec 10 04:19:53 ossim sshd[11366]: Failed password for root from 10.0.1.3 port 64270 ssh2"
username="root" 

2007-12-10 04:20:06 OSSIM-Message: Event received: event id="0" alarm="0" type="detector"
date="2007-12-10 04:20:05" plugin_id="4003" plugin_sid="7" src_ip="10.0.1.3" src_port="64270"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" asset_src="1" asset_dst="1"
log="Dec 10 04:20:05 ossim sshd[11366]: Accepted password for root from 10.0.1.3 port 64270 ssh2"
username="root" 

2007-12-10 04:20:06 OSSIM-Message: Event received: event id="3" alarm="0" type="detector"
date="2007-12-10 04:20:06" plugin_id="1505" plugin_sid="21" src_ip="10.0.1.3" src_port="64270"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10"
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 1, Priority: 5
Rule 1 [2007-12-10 04:20:00] [4003:1] [Rel: 7] 10.0.1.3:64270 -&gt; 0.0.0.0:0
Rule 2 [2007-12-10 04:20:06] [4003:7] [Rel: 10] 10.0.1.3:64270 -&gt; 0.0.0.0:0
" 

2007-12-10 04:20:07 OSSIM-Message: Event received: event id="4" alarm="0" type="detector"
date="2007-12-10 04:20:06" plugin_id="1505" plugin_sid="22" src_ip="10.0.1.3" src_port="64270"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10"
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 2, Priority: 5
Rule 1 [2007-12-10 04:20:01] [4003:1] [Rel: 7] 10.0.1.3:64270 -&gt; 0.0.0.0:0
Rule 2 [2007-12-10 04:20:06] [4003:7] [Rel: 10] 10.0.1.3:64270 -&gt; 0.0.0.0:0

Explanation

Here we wrote two directives with exactly the same events matching, naming it differently. That way we’ll be able to answer the concurrency question since I honestly don’t remember how that part worked.

As you can see in the logs, two events arrived, and two directives got matched.

Result: two alarms

(Image removed, broken link, I’m very sorry. DK.)

Case 2: Two concurrent correlation rules, a third grouping rule

Directives

&lt;directive id="21" name="Testing concurrency 1" priority="5"&gt;
  &lt;rule type="detector" name="Level 1:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" &gt;
        &lt;rules&gt;
                &lt;rule type="detector" name="Level 1:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" /&gt;
        &lt;/rules&gt;
  &lt;/rule&gt;
&lt;/directive&gt;

&lt;directive id="22" name="Testing concurrency 2" priority="5"&gt;
  &lt;rule type="detector" name="Level 2:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" &gt;
        &lt;rules&gt;
                &lt;rule type="detector" name="Level 2:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" /&gt;
        &lt;/rules&gt;
  &lt;/rule&gt;
&lt;/directive&gt;

&lt;directive id="23" name="Aggregated test directive" priority="5"&gt;
  &lt;rule type="detector" name="Directive 1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" <strong>plugin_id="1505" plugin_sid="21,22"</strong> &gt;
        &lt;rules&gt;
                &lt;rule type="detector" name="Directive 2" reliability="10"
                occurrence="1" from="ANY" time_out="300" to="ANY" port_from="ANY" port_to="ANY"
                <strong>plugin_id="1505" plugin_sid="21,22" sticky="true" sticky_different="SRC_IP"</strong> /&gt;
        &lt;/rules&gt;
  &lt;/rule&gt;
&lt;/directive&gt;

Logs

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="0" alarm="0" type="detector"
date="2007-12-10 06:24:55" plugin_id="4003" plugin_sid="7" src_ip="10.0.1.3" src_port="64620"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" asset_src="1" asset_dst="1"
log="Dec 10 06:24:55 ossim sshd[19967]: Accepted password for root from 10.0.1.3 port 64620 ssh2"
username="root" 

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="35" alarm="0" type="detector"
date="2007-12-10 06:24:56" plugin_id="1505" plugin_sid="21" src_ip="10.0.1.3" src_port="64620"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10"
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 1, Priority: 5
Rule 1 [2007-12-10 06:24:51] [4003:1] [Rel: 7] 10.0.1.3:64620 -&gt; 0.0.0.0:0
Rule 2 [2007-12-10 06:24:56] [4003:7] [Rel: 10] 10.0.1.3:64620 -&gt; 0.0.0.0:0
" 

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="36" alarm="0" type="detector"
date="2007-12-10 06:24:56" plugin_id="1505" plugin_sid="22" src_ip="10.0.1.3" src_port="64620"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10"
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 2, Priority: 5
Rule 1 [2007-12-10 06:24:51] [4003:1] [Rel: 7] 10.0.1.3:64620 -&gt; 0.0.0.0:0
Rule 2 [2007-12-10 06:24:56] [4003:7] [Rel: 10] 10.0.1.3:64620 -&gt; 0.0.0.0:0
" 

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="37" alarm="0" type="detector"
date="2007-12-10 06:24:56" plugin_id="1505" plugin_sid="23" src_ip="10.0.1.3" src_port="64620"
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10"
asset_src="1" asset_dst="1" data="directive_event: Aggregated test directive, Priority: 5
Rule 1 [2007-12-10 06:24:56] [1505:21] [Rel: 3] 10.0.1.3:64620 -&gt; 0.0.0.0:0
Rule 2 [2007-12-10 06:24:56] [1505:22] [Rel: 10] 10.0.1.3:64620 -&gt; 0.0.0.0:0

Explanation

This is exactly the same event as before with two added features: a grouping directive with our two recently created directives and a couple of keywords that prevent our grouping directive to fire twice, since the same event is used for level 1 and level 2. Using sticky=”true” and sticky_different=”SRC_IP” we are telling the engine to not spawn a second instance of the directive if it comes from the same source IP.

I think everything is quite self-explanatory (having read the other correlation docs avaiable at the ossim.net docs section).

Result: one alarm, grouping two others

(Image removed, broken link, I’m very sorry. DK.)

Foot note

I’ve detected some glitches during these tests, mainly in the alarm panel grouping code. As you can see on the screenshots event count isn’t correctly interpreted and separate alarms are being spawned, besides the grouped one. I’ll check that asap.

DK

Mr Wolf Wannabe.

More Posts - Website

, , ,

Tutorial 3: First recommended steps after installation

December 7th, 2007 | Posted by DK in Tutorials - (1 Comments)

This tutorial tries to show the first common steps you could perform if you’re new to ossim and just finished installation, without knowing what to do next.

The tutorial will cover:

  • Policies
  • Initial Inventory
  • Scans
  • Scheduled scans
  • What to do next

Many topics we’ll cover on this tutorial can be extended checking the documentation wiki.

BREAK

Asset definition: the quick way

Ok, first we should populate our asset database a bit. I’m pretty sure I know my home network/testing environment pretty well, but this could change on a larger scale network or an unknown environment.

The first thing I’ll do is to ask the administrator (that is, myself) about the networks we’ve got and which are directly or indirectly seen by ossim.

The answer is two networks:

  • Public/internal network, connected to the gateway. It’s a class C network in the 192.168.1.x range.
  • Private/internal network, the one served by my AP. It’s a class C network too but since dhcp assigns addresses starting at 10.0.1.2 and I’ve got only a couple of hosts I’ll choose a smaller network (/28)

Networks

So let’s start by adding those networks with a default priority and rrd profile, without nessus scanning activated:
(Image removed, broken link, I’m very sorry. DK.)

Networks are very important and you should define as many as you’ve got, even if ossim is not directly connected to them. This is because hosts defined in DB or belonging to a network are being treated differently and more information for them is being used (Passive OS and Service information only gets saved for defined hosts/hosts belonging to defined networks).

Network groups

Network groups are being used to aggregate information in case you’ve got many networks. In my case it’s ridiculous, but we’ve got customers with hundreds of networks where they couldn’t live without this feature :-)

You can group by any criteria you want, either division, building, network ranges, whatever:
(Image removed, broken link, I’m very sorry. DK.)

In this case we’re raising the threshold a bit since aggregated networks are supposed to raise more events than individual hosts / networks. 100 may be too low, but it’s a matter of practice / experience and varies for each environment.
Group information is specially useful for risk related panels, such as aggregated risk and riskmeter, preventing your pages from growing too much. Individual networks will be highlighted if their compromise or attack gets over their threshold:
(Image removed, broken link, I’m very sorry. DK.)
Hint: If you click on the colored area of the risk graph you’ll be redirected to the hosts involved in that risk peak. If you click on the host you’ll get redirected to the event listing that has created that risk situation.
(Image removed, broken link, I’m very sorry. DK.)

Netscan

So, next we would like to define a couple of hosts in order to personalize their asset information and to run reports and scans on them, and not on the entire network.

Hosts can be inserted manually as we’ll see on the next point, but it’s much easier to let ossim scan them using nmap and give some default data for all of them to be inventoried. Again my example is a bit short on hosts (only two), but think about a bunch of class C networks and the work you could save.

So we get to tools->netscan, select our private network and voila:
(Image removed, broken link, I’m very sorry. DK.)

Note: depending on the size of your network this may take a while, and an php timeout would make this page unusable. You might want to increase php’s max data size and session expiration.

Hosts

Since I don’t really want to wait for a scan on my other class C network I’ll insert them manually, as seen on the next images.
(Image removed, broken link, I’m very sorry. DK.)

As an added benefit, you’ll be able to see all the collected information about each host you define here:
(Image removed, broken link, I’m very sorry. DK.)

Groups

Now we’ll do our first nessus scan. Let’s scan only the workstations, leaving the gateways untouched. We could enable them one by one, yes, but again that can get tiring for many hosts. The ideal solution is to create groups (you can create as many as you wish, having hosts in different groups too) so you can later on apply separate scan types and schedules on them.

Let’s create a simple one, as said, only workstations: (we’ll use it later)
(Image removed, broken link, I’m very sorry. DK.)

OCS Inventory

Having extended inventory information from your hosts is also useful. See the old tutorial on OCS for more information on how to setup OCS. Hint: execute the installer as part of your logon scripts on a domain, that way you’ll get a lightning fast inventory of your whole network.

Nessus Scan

Using the group we created before, we’ll check how vulnerable my workstations are.
(Image removed, broken link, I’m very sorry. DK.)

Unf, 8. That’s not much, but taking into account that each vulnerability can have a level ranging from 0 to 10, I should look at the reports by clicking on the host.
Aaah, almost forgot it. Since my vulnerability_incident_threshold at configuration->main is set to “0″, every vulnerability with level 0 or higher will create a new incident too, as we can see here:
(Image removed, broken link, I’m very sorry. DK.)

Note: Vulnerability incidents are able to handle false positives too. If you close it and tag it as false positive, it won’t be opened on the next scan. But be warned, if you close it and it’s not tagged, it will get opened again next time. That way you’ll can happily close the incident when your sysadmin says he’s patched the hole, knowing that if he has lied he’ll have that incident on the table next month again :-)

(Image removed, broken link, I’m very sorry. DK.)

Hint: you can check to see if the scan is actually working by grepping on the agent, since the information shown during scan ain’t very verbose.

Hint2: Nessus scan is distributed, that is, ossim tries to find the sensor associated to the target in order to scan from there.

ossim:~# ps ax | grep nessus
 2536 ?        Ss     0:00 nessusd: waiting for incoming connections
 6600 ?        S      0:00 /usr/bin/nessus -c /usr/share/ossim/www/vulnmeter/tmp/.nessusrc
-x -T nsr -q 10.0.1.50 1241
/usr/share/ossim/www/vulnmeter/tmp/sensors/10.0.1.50.targets.txt
/usr/share/ossim/www/vulnmeter/tmp/sensors/10.0.1.50.temp_res.nsr
 6606 ?        Ss     0:03 nessusd: serving 10.0.1.50
 6614 ?        S      0:00 nessusd: testing 10.0.1.3
 6615 ?        S      0:00 nessusd: testing 192.168.1.33
 6618 ?        S      0:00 nessusd: testing 192.168.1.33 (/var/lib/nessus/plugins/nessus_tcp_scanner.nes)
 6619 ?        S      0:00 nessusd: testing 10.0.1.3 (/var/lib/nessus/plugins/nessus_tcp_scanner.nes)
 6625 pts/0    R+

Schedule scans

Finally, let’s leave a scan scheduled to execute once a month:
(Image removed, broken link, I’m very sorry. DK.)

Final words

That’s all for today, it may shed a bit of light on some easy to use, basic features of ossim. If you followed these steps you can be pretty sure that you’ll start getting more and more useful information out of the system.

Check the monitors, reports and dashboards and you’ll see how your asset information starts to affect all the indicators.
(Image removed, broken link, I’m very sorry. DK.)

The next thing I’d recommend is checking out the tools at “Tools->Downloads” and install ossec and ocs, in order to get even more information out of your network’s hosts.
(Image removed, broken link, I’m very sorry. DK.)

DK

Mr Wolf Wannabe.

More Posts - Website

, ,

Tutorial 2: Syslog data mining with attached md5sum. AKA “Store 100% of data”.

December 6th, 2007 | Posted by DK in Tutorials - (Comments Off)

1. The need. The Hype.

There’s obviously a need for storing vast amount of logs, and few things today aren’t able to log into syslog. So it’s just obvious to stumble upon that request every once in a while, and this tutorial illustrates the OSSIM approach at massive syslog data storage. Of course, where you say syslog you can say windows event log, snmp data, whatever generates a big amount of raw data.

Compliance

I don’t know much yet about all of this compliance stuff (I were lucky, Julio always has been much more knowledgeable on that area than me so I could skip it) but I guess I’ll have to start learning, there are just too many people asking for it and I’m getting very curious.

From what I’ve seen, a short list of regulations requiring, or at least strongly recommending a certain amount of raw data storage and reports are:

  • ISO27001/17799
  • SOX
  • HIPAA
  • PCI
  • Basel II
  • NIST 800-53
  • Many more…

(Searching for SIM and compliance information I see that’s a major marketing point from vendors too, well, just for the records, ossim helps you to be compliant with all that stuff)

Centralized logging

Maybe the need is pure sysadmin’s lazyness. You want to be able to answer to questions you get asked by your management / customers in the easiest possible way.

I heard this from a guy a couple of days ago: the more information about your network you’ve got, the more answers you can give, and that’s exactly what SIM/SEM systems are good at.

Data mining

This is a bit redundant with the previous entry, but there are people that just don’t care about exact data, but they’re in desperate need of colorful graphs in order to be able to keep their bosses calm. Well, having logs from everything in your network allows for easy colorful report generation with little knowledge of the underlying data. The worthyness of those reports in the end will be highly questionable of course.

BREAK

2. The preparation.

I thought it would be interesting to explain the process we used to create this plugin in the first time here, so it will be a hybrid tutorial: syslog and how to fetch your own datasources.

Write a new function

Well, the first thing that came to my mind was that, even without knowing much about those compliance regulations, if they’ve been designed with at least a bit of common sense they’ll require a digital cryptographic stamp on the logs. So let’s just add a little function to our plugin environment that does just that: timestamping using md5.

def md5sum(datastring):
    return md5.new(datastring).hexdigest();

Just put your functions into ParserUtils.py and they’re available for all plugins. That file usually resides at /usr/share/ossim-agent/ossim_agent/.

Write a new plugin

If you have no clue about ossim plugins I’d recommend checking this wiki entry.

The next thing we’ll do is create a new plugin. We want this plugin to:

  • Get every single syslog line
  • Respect the orignal logline
  • Checksum that line

After a bit of toying I’ve seen it would be easy to also:

  • Extract the sensor from the line
  • Extract the source ip
  • Extract the originating process
  • Extract the PID
  • Extract the line without the “changing” part

All of this while respecting the original line of course.

;; syslog
;; plugin_id: 4007
;; type: detector
;;

[DEFAULT]
plugin_id=4007

[config]
type=detector
enable=yes

source=log
# Enable syslog to log everything to one file. Add it to log rotation also.
# echo "*.*     /var/log/all.log" &gt;&gt; /etc/syslog.conf; killall -HUP syslogd
location=/var/log/all.log

# create log file if it does not exists,
# otherwise stop processing this plugin
create_file=true

process=
start=no
stop=no
startup=
shutdown=

## rules

[syslog - datamining]
# Sep  6 12:07:26 ossim-devel su[9886]: FAILED su for root by juanma
event_type=event
regexp="^(?P(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(?P[^\s]+)\s+(?P<source />[^\s]+)\s+(?P[^\[]*)\[(?P
\d+)\]:(?P.*))$"
sensor={resolv($sensor)}
date={normalize_date($1)}
plugin_sid=1
sensor={resolv($sensor)}
userdata1={md5sum($logline)}
userdata2={$logline}
userdata3={$generator}
userdata4={$logged_event}
userdata5={$pid}

This is quite a simple plugin. Only one rule, a relatively simple regexp, you may notice the md5sum function being used, as well as a couple of others.

We don’t really need to extract the fields here but that will make getting reporting done for this plugin a very very easy task.

Write the SQL

Next thing we have to do is make the server aware of our new plugin. We’ll assign it ID 4007 since it’s free and it’s in the range reserved for syslog, and insert only one event.

Please notice this event is being inserted with priority 0, more on this later.

DELETE FROM plugin WHERE id = "4007";
DELETE FROM plugin_sid where plugin_id = "4007";
INSERT INTO plugin (id, type, name, description)
VALUES (4007, 1, 'syslog', 'Syslog plugin with md5 checksum logging');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability)
VALUES (4007, 1, NULL, NULL, 'Syslog: syslog entry' , 0, 1);

Priority 0

Inserting the event with priority 0 is a little trick that helps us preventing these items from generating noise to our SIM system.

By default, each incoming event as an instant risk value calculated as:

risk = asset * priority * reliability / 25

More information can be found on our wiki.

So, what would happen if this plugin is activated and you enable debugging on an app that generates a thousand log lines ? Well, that host would be treated as compromised, alarms would raise, and many indicators would be affected by this false positive.

Since we actually only want to store events, not assess any instant risk with them, one of three variables on the multiplication should be set to 0 for this type of events to be ignored by the risk assessment system by default (they would still be correlated, forwarded, could become alarms, etc. etc…). We can’t control the assets, and the event’s reliability can only be modified by correlation rules. Wouldn’t it be nice to be able to decide which events to take into account or not based on our policy ? well, the obvious choice is priority, since correlation rules can affect both priority as well as reliability and policies only affect priorities.

That means if we want specific source ips or generators to behave differently, we can control it via policy, and if we want to generate alarms, correlate with other events or based on some of the extracted fields, we do it using directives (correlation rules).

Enable syslog logging

There are better ways of getting all the logs into one file without duplicating but just for the records sake, I added “*.* /var/log/all.log” to /etc/syslog.conf and restarted the service. That way we can point our plugin at that file and forget about filtering.

Enable remote logging

Remote logging is similar, but entering “*.* @agent_ip” instead into the syslog file and restarting. Here is my output on MacosX:

Gestalt:etc dk$ sudo vi syslog.conf
Gestalt:etc dk$ ps ax | grep syslog
   13   ??  Ss     0:04.84 /usr/sbin/syslogd
 3994 s004  R+     0:00.00 grep syslog
Gestalt:etc dk$ sudo kill -HUP 13
Gestalt:etc dk$ syslog -s test

On our <em>tail -f /var/log/all.log</em>:

Dec  6 07:10:12 10.0.1.3 Gestalt syslog[3997]: test

3. The implementation.

First of all, if you want to reproduce it, get the plugin and SQL files from here.

Update 2008/03/11: Fixed the regexp, as can be seen in comments below it was too narrow. Just re-download the .cfg.txt

Copy plugin to it’s place

The plugin file should be put into /etc/ossim/agent/plugins/syslog.cfg.

Insert SQL

The SQL can be inserted from the server as:

cat syslog.sql.txt | mysql -p -uroot ossim

Note to the installer users: you can get your database password from /etc/ossim/ossim_setup.conf as root, grepping for “pass” since it gets generated at install randomly for each installation. That password is also used for ntop, nessus, etc…

Restart server

There are nicer ways but this works:

killall ossim-server; ossim-server -d

Enable plugin

Add a line like this to your /etc/ossim/agent/config.cfg, into the [plugins] section.

syslog=/etc/ossim/agent/plugins/syslog.cfg

Restart agent

killall ossim-agent; ossim-agent -d

Does it work ? RT event viewer.

The best way to see if an event traverses through generator –> agent –> server –> database is to fire up the RealTime event viewer and start it selecting “fast”.

Here you can see they’re starting to arrive:
(Image removed, broken link, I’m very sorry. DK.)

4. The results.

That’s it, the data is being fed into the database. What now ?

Policy

First of all, remember that the event’s won’t affect our risk but they’re still getting stored.

WARNING: If you abuse this, your database won’t be able to handle the load from all that data. You’re going to store it, but with little use. Default OSSIM is not tuned for more than 1, 1.5 Million active events in database. That’s more than enough for a small/home user, but way too much if you feed it with 2000 syslogging devices.

So what could you do ? store only part of the data using policies when you need it, correlating all events for more interesting stuff. This obviously doesn’t help if you came here looking for better compliance. (See below, sections 5 and 6).

Following are a couple of hints about how to create a policy that only stores events from a single syslog host.

  1. Create a host: (Image removed, broken link, I’m very sorry. DK.)
  2. Create a plugin group with syslog plugins: (Image removed, broken link, I’m very sorry. DK.)
  3. Select the generating host as source (you could also apply this to a sensor, and any hosts it receives) (Image removed, broken link, I’m very sorry. DK.)
  4. Disable everything, just store (or leave correlation enabled for these events if you wish) (Image removed, broken link, I’m very sorry. DK.)
  5. Create a drop-everything else policy: (Image removed, broken link, I’m very sorry. DK.)
  6. The result: (Image removed, broken link, I’m very sorry. DK.)

Performance

I’ll just throw in the warning in again:

WARNING: If you abuse this, your database won’t be able to handle the load from all that data. You’re going to store it, but with little use. Default OSSIM is not tuned for more than 1, 1.5 Million active events in database. That’s more than enough for a small/home user, but way too much if you feed it with 2000 syslogging devices.

Which basically means: if you’ve got few devices (at most 200000 events a day) you can go for “logging everything” with the default installer ossim. Otherwise you’ll have to heavily tune your database, enable “writing into filesystem” and/or play some other DB tricks. See section 6 below.

Event viewer

Since we’ve already separated our info into bits, besides of keeping the whole original line, we can easily create a new event viewer panel for this specific plugin.

In this example below, I created one with four columns, and label:data relations such as:

  • DATE:DATE
  • Checksum:USERDATA1
  • Generator:USERDATA3
  • Log:USERDATA4

(Image removed, broken link, I’m very sorry. DK.)

Note: log is not the entire line, that would be USERDATA2 as you can see on our plugin.

Reports

What else do we need to be able to tell our boss that we’re now much closer to being compliant ? nifty graphs :-)

Let’s define a new tab (using a nice icon from images.google.com) and put two panels in there, a simple graph with our top 5 generated logs and a cloud with all the generators.

Tab: (Image removed, broken link, I’m very sorry. DK.)

Cloud definition: (Image removed, broken link, I’m very sorry. DK.)

Cloud SQL: (Image removed, broken link, I’m very sorry. DK.)

Resulting graph: (Image removed, broken link, I’m very sorry. DK.)

Note: The pie graph is of type “SQL”, using rows as labels, 45 degree rotation. Here is the export you can import into your panel:

plugin_custom_sql::
YTo0OntzOjY6InBsdWdpbiI7czoyMjoicGx
1Z2luX2NvbmZpZ19leGNoYW5nZSI7czoxMT
oicGx1Z2luX29wdHMiO2E6Mjc6e3M6ODoiZ
3JhcGhfZGIiO3M6NToic25vcnQiO3M6OToi
Z3JhcGhfc3FsIjtzOjIxOToic2VsZWN0IHV
zZXJkYXRhNCwgY291bnQoKikgYXMgbnVtIG
Zyb20gZXh0cmFfZGF0YSwgb3NzaW1fZXZlb
nQgd2hlcmUgb3NzaW1fZXZlbnQuc2lkID0g
ZXh0cmFfZGF0YS5zaWQgYW5kIG9zc2ltX2V
2ZW50LmNpZCA9IGV4dHJhX2RhdGEuY2lkIG
FuZCBvc3NpbV9ldmVudC5wbHVnaW5faWQgP
SA0MDA3IGdyb3VwIGJ5IHVzZXJkYXRhNCBv
cmRlciBieSBudW0gZGVzYyBsaW1pdCA1Owo
KIjtzOjExOiJncmFwaF90aXRsZSI7czoyMj
oiVG9wIDUgc3lzbG9nIHByb2Nlc3NlcyI7c
zoxMDoiZ3JhcGhfdHlwZSI7czozOiJwaWUi
O3M6MTg6ImdyYXBoX2xlZ2VuZF9maWVsZCI
7czozOiJyb3ciO3M6MTY6ImdyYXBoX3Bsb3
RzaGFkb3ciO3M6MToiMCI7czoxNToiZ3Jhc
GhfcGllX3RoZW1lIjtzOjU6IndhdGVyIjtz
OjE3OiJncmFwaF9waWVfM2RhbmdsZSI7czo
yOiI0NSI7czoxNzoiZ3JhcGhfcGllX2V4cG
xvZGUiO3M6NDoibm9uZSI7czoyMToiZ3Jhc
GhfcGllX2V4cGxvZGVfcG9zIjtzOjE6IjEi
O3M6MjI6ImdyYXBoX3BpZV9hbnRpYWxpYXN
pbmciO3M6MToiMSI7czoxNjoiZ3JhcGhfcG
llX2NlbnRlciI7czozOiIwLjIiO3M6MTg6I
mdyYXBoX3BvaW50X2xlZ2VuZCI7czowOiIi
O3M6MTc6ImdyYXBoX3Nob3dfdmFsdWVzIjt
zOjE6IjEiO3M6MTE6ImdyYXBoX2NvbG9yIj
tzOjc6IiMwMDAwODAiO3M6MTQ6ImdyYXBoX
2dyYWRpZW50IjtzOjE6IjAiO3M6MTA6Imdy
YXBoX2xpbmsiO3M6MDoiIjtzOjE2OiJncmF
waF9yYWRhcl9maWxsIjtzOjE6IjEiO3M6MT
E6ImdyYXBoX3lfbWluIjtzOjE6IjAiO3M6M
TE6ImdyYXBoX3lfbWF4IjtzOjE6IjAiO3M6
MTE6ImdyYXBoX3hfbWluIjtzOjE6IjAiO3M
6MTE6ImdyYXBoX3hfbWF4IjtzOjE6IjAiO3
M6MTE6ImdyYXBoX3lfdG9wIjtzOjE6IjAiO
3M6MTE6ImdyYXBoX3lfYm90IjtzOjE6IjAi
O3M6MTE6ImdyYXBoX3hfdG9wIjtzOjE6IjA
iO3M6MTE6ImdyYXBoX3hfYm90IjtzOjE6Ij
AiO3M6MTU6ImV4cG9ydGVkX3BsdWdpbiI7c
zoxNzoicGx1Z2luX2N1c3RvbV9zcWwiO31z
OjExOiJ3aW5kb3dfb3B0cyI7YTozOntzOjI
6ImlkIjtzOjM6IjF4MSI7czo1OiJ0aXRsZS
I7czoxMDoiVG9wIDUgbG9ncyI7czo0OiJoZ
WxwIjtzOjA6IiI7fXM6MTE6Im1ldHJpY19v
cHRzIjthOjQ6e3M6MTQ6ImVuYWJsZV9tZXR
yaWNzIjtzOjE6IjAiO3M6MTA6Im1ldHJpY1
9zcWwiO3M6OToic2VsZWN0IDQ7IjtzOjEzO
iJsb3dfdGhyZXNob2xkIjtzOjE6IjMiO3M6
MTQ6ImhpZ2hfdGhyZXNob2xkIjtzOjE6IjU
iO319

5. The tuning/solutions.

After the two warnings, there are some solutions for the problems.

Database rotation

First we implement database rotation every backup_day days. By default this is set to 5, so if you are collecting a maximum of 200.000-300.000 syslog events a day, you should be fine. If that number is much more, you can increase the days, if it’s higher, decrease. That’s the quick and dirty solution.

File storage

A friend of mine is working on filesystem storage in the form of /date/sensor/plugin/ip.log, hooks are there for that format and it shoud be available very soon.

6. The spam.

I really hate spamming on a community article, please don’t read any further if you’re satisfied with what you’ve got so far and don’t have any special requirements.

Compliance module: Performance, storage, reporting.

The founders of ossim have created a company about a year ago, called (in order to be original) ossim. The site is ossim.com and there we provide a series of added value solutions and services.

Since all of this compliance stuff is really a high level enterprise need we’ve spent a lot of time packaging a few custom modules just for that matter, and are continously improving them in parallel with normal ossim evolution.

So the compliance modules are being provided as sort of an add-on to the open ossim distribution, since we feel that that’s a part useless for the average community user and we would like the companies that are saving many hundres of thousands of dollars by using our opensource solution to put something back into development.

Professional compliance modules include:

  • Appliance version for easy deployment
    • Heavily tuned database, using a combination of Heap, Merge, InnoDB, MyISAM and compressed MyISAM tables for optimal performance / storage capacity.
    • 64Bit version
    • MMAP compiled libpcap applications for greatly enhanced performance (Up to Gigabyte speeds with a standard dual core machine)
    • Sets of custom flash graphs for specific compliance needs
    • Sets of custom viewers for specific compliance needs
  • Guides for particular compliance needs. “What exactly do I have to setup in ossim to be more compliant with XXXX”
  • Specific support
  • Specific compliance rule feeds

DK

Mr Wolf Wannabe.

More Posts - Website

, , , , ,