Wednesday 28 May 2014

Cyber Threat Intelligence...my attempt at it...

*** update to this post: http://infosecscapes.blogspot.com/2014/06/threatelligence.html

I recently started playing with various open source threat intelligence frameworks. One of which is the collective intelligence framework (CIF):


  • https://code.google.com/p/collective-intelligence-framework/

I am not going to go to much in to the technical bits but the CIF tool collects data from various open source intelligence feeds which include data such as:
  • IP Addresses
  • Domains
  • MD5 - Suspicious malware 
  • etc. theres more to it
The solution works really well but is limited on the interface side of things, but is really good at collecting, normalising data, sharing data, feeding data to appliances (FW/proxies/snort). I like it and will continue using it. Also the data is not really that rich in information

But I decided to play around and make my own threat intelligence solution type of thing. Since I started playing with Python recently I thought it was a good start. I wrote a threat feed collector that will collect data from the same/similar sources as the CIF tool does. I made it easy and generic enough to be able to add new sources as I find them.

It was a success I am collecting this data really well, but I was at a loss of how to store it, view it, make it useful...I was up all night thinking about a database structure and how not to make it horrible...

Then it clicked to me, intelligence data should not be structured, and so I finally had a reason to play with ElasticSearch:


  • http://www.elasticsearch.org/
In short (because I am only scratching at the surface), ElasticSearch is a fast, schema-less "database". I found it to be excellent at storing large data quickly and efficiently. You must really go and research it. I started with a simple article and took it from there:


  • http://joelabrahamsson.com/elasticsearch-101/

So now I am able to collect a lot of data and easily store it and access it. I took it further by enriching the data that I am collecting by looking up country, geo locations etc.


My next problem was a big one that I thought I would be spending the most time on. Which was the interface to view the data. I was thinking about the mammoth task of creating an interface that will make the data searchable, easy to view, and make it informative and eye candy friendly

....another problem solved by the ElasticSearch team, namely Kibana which recently became apart of ElasticSearch it seems. 
  • http://www.elasticsearch.org/overview/kibana/

Kibana essentially is an Dashboard interface for querying ElasticSearch data and presenting it beautifully. Within no time I was building really pretty dashboards using Kibana.

Here are some examples I built...by the way I did this all over 2 days from start to where it is now! Below you can search for the entire database for specific key data...and its fast!





The next steps are to automate the data collection, add a limit to the amount of days to keep data (you want to stay relevant and not sit with old data, especially when it comes to cyber threats.) and even add some sort of alerting.

I started the project on a saturday and got to this point by sunday and even got to spend time with the family. 

Python + Elasticsearch + Kibana = Threatelligence for the win!

I can now focus on collecting data, normalising + correlating and enriching it further...all in an automated fashion. I will push a release to Github soon.




 

Sunday 11 May 2014

Tabs over Curly brackets

Python over Perl...
 
Last week I was talking to an infosec guy from Australia, and we briefly spoke about programming/scripting languages that we were comfortable in. I mentioned that I am comfortable with C#, bash, PHP, Perl etc. He mentioned that a few guys at his office are into Python and I replied with...."who programs in tabs it makes no sense...curly brackets for the win!" He agreed with me and couldn't understand the whole thing about tabs too...

I mean really, tabs...tabs!

Anyways it got me thinking, perhaps I should try it before I diss it! I usually need some sort of motivation to start something new, especially learning/adapting to a new programming language. So I remembered a few years back, when I worked at SensePost, at the time Roelof Temmingh was working on a research project named Setiri which he demo'ed at BlackHat. Setiri was a smart trojan at the time, it pretty much opened an invisible Internet Explorer which it hooked and used it to communicate to the Command & Controller server. It bypassed all the firewall rules and beat proxies etc. because hey it was IE and IE was allowed out...

Here's a link to a BH presentation on Setiri, click

So I set the challenge to hard and figured if I can do a little something similar in Python then maybe it would grow on me...that is...if Python could even can do it.

I installed the latest version Python (3.4) and started Googling and putting a few lines of code together...and well before I knew it...I was mostly done...and I was only 2 hours into my freshly installed version of Python...surprised...yes! disappointed...hell no!

I am going to say that I am impressed, the original Setiri Trojan was about give or take 1,000 lines of CPP, I managed a Python version which works in the exact same way in about 65 lines of code.

I am not going to call it Trojan, because I do not write them...I am interested in them yes...I like to poke at malware and understand it but just not my cup of tea...so I am going to call it an agent (: And whats really awesome is that I can bundle everything into a standalone EXE (Py2Exe/PyInstaller/PwnInstaller)...so it runs on Windows even when Python is not installed on the Windows machine.

Opening up an  invisible Internet Explorer is a simple module import and 2 lines of code, below is a sample with one or two more lines:


So obviously my version has a bit more to it...but Python really is that simple, for years people have been telling me whatever you can do in Perl or whatever other langauge you can do it in a lot less lines of Python...and I blew them off back then.

Here is just a sample of opening up Internet Explorer in CPP, 5 lines of code before you even browsed off to the command server:

 

Its so easy...everything is really much easier in Python, I have spent the weekend on and off in my spare time with Python and I am loving it...WHY HAS IT TAKING ME SO LONG?!?!

So I have a full on working agent and server (PHP) which runs really well, but it has spawned off a few more idea's for a testing tool (: So I have some more learning and playing to do! More on that in my next post...when I have something to show...






Sunday 4 May 2014

oneVault - Updates


Updates on the blog about oneVault have have been few but work on bugs and feature requests have been recently working out well due to a number public holidays and long weekends giving me some time to dev.

There have been a number of bug fixes and speed improvements, but one of the new features is a pre-project scoping section. Someone asked me about getting oneVault involved before the project begins, specifically about pre-project engagement...scoping.

This is sort of my take on it...

Managers can now create dynamic project scope questionnaire templates which can be used over and over again, in the below screenshot we have a basic scope questionnaire template, easily designed in minutes. You can design questions with text, single choice or multiple choice answers, whatever suites your requirement:


When a new pentest requirement comes in the manager can select a scoping questionnaire template and send it off to the client. oneVault will send an email off to the end-client with a dynamic link to the questionnaire which the client will fill out:



The manager will be able to keep an eye on the questionnaires sent out and review completed applications


From here the Manager will be able to create a new project, assign it to a pentester and they will have access to view the scope and work accordingly within the scope.

I am pretty happy with it and its all coming along quite nicely. If you want to beta or play with it get hold of me or send me requests and idea's you would like to see in it.