Project 4 - Web based visualization for malware/attack analysis

Primary mentor: Ben Reardon (AU)
Student: Oguz Yarimtepe

Project Overview:
Creating a 3D Earth Visualization, that will show malware/attack analysis on a time line basis with heat map tiles and mesh structure.

Project Links:
The current status of the project can be viewed from the documentation site: http://webviz.comu.edu.tr/doc

The site also includes links to the demo sites with installation instructions.

Project Plan:

  • May 13-May23: Dionaea sqllite data investigating, digging the collected data by the supplied Python scripts, also it is better to spend some time for the RabbitMq and Orbit usage design for the live data pushing to the browser. It will be a good start if i decide how to use these technologies before the coding starts.
  • May 24-June 11: Finishing the live data pushing by using RabbitMq and Orbit
  • June 15-July 11: By using GeoDjango, creating a view with the Google Earth KML. Creating mesh tiles on this Google Earth map, which has the z value showing the number of attacks or malware. A grid structure will be drawn over the map with a resolution as a variable (a square of n km) and each attacker latitude and long within that square would add to "z" of that square. The z value will increase each time when the attack is in range. The z index color will change according to the height. The more higher, means the more attack is detected and the color will have more red, else the greener tiles will be shown. The resolution should be changeable.
  • July 11: Submitting for mid-term evaluation
  • July 12-August 14: Add animated view for the visualization, colored grid mesh will be created over time
  • August 15: Documentation, review time (better to use docstrings while coding in a pep-8 style)
  • August 22: Final submitting

Blog:

There is a blog address that includes progress and updates about the project. Follow it from here

Update:

29 May 2011:
Last week spent for visualizing the Dionaea sqlite data with WebGL Globe. A Django project is created. A sample page is written that is pulling the data from sqlite via sqlite3 Python binding. At the first week of the coding phase, a pushing sample was created, that pushes data from sqlite to browser.
Next: Integration with the pushing and WebGL Globe, also i should test the visualization with a larger data set.
After: I should be starting to work on the 2D map with mesh tiles.
6 June 2011:
Currently two globe view is implemented. One is statically displaying the attacks according to regions, the other one should be working via push method. Both are nearly same. hpfeeds' feed.py is changed according to save data to postgis database. By looking at the IP, geographic location is calculated and returned as json response. The other one is getting the json response in a pushed way using Orbited. It is not working correctly, either because of Orbited subdomain requirements or something else. I am running Orbited server at the same time, but some how i am not able to connect to the server from the browser side, got some cross-domain errors.
The first working prototype of the globe is done. You may check a post here
Next: Will add location and hit counts to the views and try to fix the live data issues
13 June 2011:
Proof of concept work is done. WebGL Globe visualization is deployed at the http://webviz.comu.edu.tr/globe/show address. Currently it is working with hpfeeds data statically.
Next: It is time to dive into the the mesh and heat map issue now
20 June 2011:
WebGL Globe work is improved with additional information on page. Time stamp information is added and country based hits are grouped in the right way. Toxiclibsjs is examined. Mesh examples are examined also.
Next: Create a 2D map using Google Map (KML) and grids on it.
27 June 2011:
2D Google Map is created. Toxiclibsjs seem a little messy. Instead of using the mesh classes defined at the toxiclibsjs, it is decided to use processingjs base functions and create the grid. A sample is presented with a gravity effect here.
Next: Create the grid on the map
4 July 2011:
Tried sketching over google map via ELabel. A processing.js grid is produced by using 3D vertices and combining them via lines. Unfortunately the try was unsuccessful. I learned that even Openlayers are not for 3D views. I will be also needing to change the camera view of the map to display the elevations.
Next: Tile map server and WebGL Earth custom API combination can be a solution for what i am looking for. I will be working on to sketch the processing.js visualization on the 2D map and use custom API to display it over 3D earth
11 July 2011:
Trying to create the mesh at the tileserver part was a wrong idea. I realized that i was searching for a wrong idea. I then checked the PhiloGL, whether i can use it. My mentor adviced me to forget about the tileserver solution. Suggested me to check the previous examples he showed. Also added this link to his mail: http://senchalabs.github.com/philogl/
So i decided to create the mesh with escalations.
18 July 2011:
Worked on grid mesh. Used processing.org. A sample with 3 random elevations on the grid worked. To bind this work with the hpfeeds data, XML responses are created at the web page side. A hypothetical application is written, taking projected x and y coordinates from the Django XML response with the magnitude values and creating mesh.

Update on 20 July:

By using XML responses from Django side a mesh is created using processing.org.

Mesh grid displaying top 5 malware sources as grid

Currently displaying top 5 malware sources with their names. Theres is a camera view that can be changed with the mouse interaction. I had problems when i exported it as applet. Will be working on it.

Next: Test the application, make it run. Deploy it on the web.

25 July 2011:
Mesh application created (check the post on related with 20 July, above) and deployed on web. Check it via Google Chrome.

Next: Heat map creation and map deployment to the base of the mesh

1 August 2011:
Base earth map is added to the mesh grid. The working is changed from country based to IP based. It is realized that there is an error at mapping latitude/longitude values to 800*600 area's x and y coordinates. The problem is most probably about using wrong conversion for the wrong map style.

I will try two methods and use the maps created by them. Miller and Mercator projection will be the candidates.

Next: Fix the x/y coordinate problem. The elevations are not fitting the right places on the map. Deploy the IP based changed applet to the web.. Start heat map, changing the colors of the tiles proportional to the magnitudes.

8 August 2011:
Base map is changed as if it will support miller projection. This change caused also fixing the x/y coordinate problem. Working is changed from country based to IP based. The elevations are created after grouping by IP numbers and mapping them to x/y coordinate system. Heatmap is created. All work is deployed on web. Legend information is also set on the same visualization.

Next: Start writing documentation for the project.

15 August 2011:
Documentation is online. Mesh base map is taken closer to the tiles a bit.

Next: Nothing to do. Wait for the final evaluation result.