Project 6 - AfterGlow Cloud

Student: Surya Nallu
Primary mentor: Raffael Marty
Backup mentor: Ralph Logan

Google Melange: http://www.google-melange.com/gsoc/project/google/gsoc2012/ayrus/8001

Project Overview:
This project aims to bring AfterGlow, an existing visualization tool to the Web (currently a command-line based tool). The project would enable AfterGlow as a service where in users can upload data and visualize their uploaded
data as graphs on-the-fly; while having the ability to control the output and its rendering options.

The existing script can be found here: http://afterglow.sourceforge.net

Project Plan:

  • April 23rd - May20th: (Goal: have a working understanding and setup):
  • Investigation: Go through the documentation of AfterCloud and GraphViz.
  • Initial set-up of resources and infrastructure setup as necessary.
  • Week of May 21st: (Goal: Design and specifications):
  • Work on a design overview – reflecting the interface and an overall gist.
  • Backend design: Concurrent sessions, UI.
  • Week of May 28th and June 4th: (Goal: General setup):
  • Work on passing an output from the Perl script to GraphViz
  • Begin implementing the server side back-end tools required for the application
  • Test this portion of the bash script
  • Week of June 11th and June 18th:
  • Begin working on the front end view (upload/input).
  • Work on the server side wrapper (with fixed inputs for this stage).
  • Extend the server side wrapper to retrieve CGI inputs from a HTML form.
  • Do testing of the server side end for different parameters.
  • Week of June 25th:
  • Create the rendering view.
  • Test the rendering with the complete application.
  • Padding time for bugs.
  • Week of July 2nd:
  • We should have the “first version” of the application by now.
  • Begin documentation of this version. Inline comments throughout the code-base will be done throughout the development.
  • Incorporate 'cleaning' of files left-over on the server (old PNG files)
  • Tasks after this week are pending review and further discussion; to be updated.

Project Deliverables:
This project will deliver a cloud service that lets users visualize data as a link graph. The service will leverage a Web application based on Django; enabling the objectives listed above. Upon successful release of the first version (by July second week) some additional add-ons will be developed to the service.

Project Source Code Repository: https://github.com/ayrus/afterglow-cloud

Student Weekly Blog: https://www.honeynet.or/blog/339

Project Useful Links:

Project Updates:

May 27th:

Completed this week:

  • Discussed and agreed upon a project design and the general functioning of the application.
  • Designed and reviewed wireframes for crurcial user front-end interfaces
  • Brainstormed scope of first version.

Planned for the next week:

  • Implement the design agreed upon earlier on Django.
  • Start witht he user UI and then proceed ahead to tieing it up with the backend.

June 3rd:

Completed this week:

  • Worked and pretty much finished on the core backend of the application. Handles some minor error, minor touch up pening.
  • Got an initial demo of the backend working.

Planned for the next week:

  • Start on the user interface end.
  • Add more error checkings to the backend, cookie support for saved configurations and document code.

June 10th:

Completed this week:

  • Cookie support enabled (for saving previous configurations). Added file cleaing for deleting old files.
  • Started on the front-end UI. Basic configurations added to the views (with proper styling).

Planned for the next week:

  • Continue working on the front-end UI and try finishing it up. Once this is done, the status should be close to finishing up the first version.

June 17th:

Completed this week:

  • Still continuing workin on the front-end UI. Implemented basic/advanced settings on the UI and tested it generally.
  • Started on adding some styling to the UI elements.

Planned for the next week:

  • Finish up the UI -- finish the UI for configurations (add the required JavaSscript to the page).
  • Add styling to the UI completely and make it look better.

June 24th:

Completed this week:

  • Finished adding all the required JS functions for configurations page.

Planned for the next week:

  • Present a demo of the current working copy and seek feedback. Fix any bugs/implement features from the feedback.
  • Add functions to incorporate deleting or changing the ordeering of the configuration lines.

July 1st:

Completed this week:

  • Added and compelted functions to change line ordering/removing from the UI.
  • Incorporated feedback from the mentor on the demo and added the changes.

Planned for the next week:

  • Finish the user interface (styling/css) according to the wire frame agreed upon in the first week -- a first version should be ready at this point.

July 9th:

Completed this week:

  • Completed work on the user interface (design) and backend for an initial release.

Live demo at this stage: http://afterglow.ayrus.net:31080/process

Planned for the next week:

  • Documentation throughout code-base for a "first" version release early in the week. (To be updated here)
  • Release the first version and begin working on the next version.
  • Plan on improvements and new features in the next version.

July 16th:

Completed this week:

  • An initial release has been made available (see links below).
  • Documentation over code-base has been completed.
  • Added a contact form with a captcha instance to the developing version

Live demo: http://afterglow.ayrus.net:31080
Source: https://github.com/ayrus/afterglow-cloud/tree/first-version
Related blog post https://www.honeynet.org/node/890

Planned for the next week:

  • Begin working on improvements to the intial version (set up parsers to export logs from different sources to CSV files; etc)
  • The timeline above will be updated over the course of the week with finalized improvements

July 23rd:

Completed this week:

  • Created a parser which accepts raw logs from any source and parses it to CSV data file (which afterglow accepts); with RegExs from the user for each column.

Planned for the next week:

  • Create pre-defined regular expressions for the parsers to use (and show how each of them would parse the log).
  • Give the ability to save regular expressions for future use.

July 30th / Aug 6th:

Completed these weeks:

  • Extended the parser mechanism - Added ability to create a regular expression to parse custom log (including an opt-in save option).
  • Users can choose predefined expressions (which contain the expressions saved by other users) with a description.
  • App administrators get an email notifying them of an expression being saved (to double check if nothing malformed has been input).

Planned for the next week:

  • Integrate with Loggly.com's API using oAuth and let users parse/render their logs from Loggly.

Aug 13th:

Completed this weeks:

  • Integrated with Loggly.com as a data source for AfterGlow. User can now access their logs uploaded on their Loggly account (using oAuth for granting access) and can then have the data parsed/rendered as a graph.
  • Added more explicit error messages when a user inputs a malformed input.

Planned for the next week:

  • Add a gallery (opt-in) so that users can submit their graphs.
  • Deploy the application
  • Documentation and cleanup for a release

Aug 19th:

Completed this weeks:

  • Added a gallery - Users can now submit their rendered graphs (if they want) to a public gallery; with the details of their graph for other users to see.
  • Deployed the application on Apache using mod_wsgi
  • Settings/configurations chosen by a user to render their graphs are remembered, so that they can choose their "Last used settings" when attempting to render again.
  • Added feature: Rendering filter for GraphViz - users can now specify whether to use neato/dot/sfdp to render ther graphs.
  • Code cleanup and refactor throughout the codebase. Added documentation in the code-base and committed a new README for a fresh install of the application. This is the second release.
  • Minor bug fixes.

A live demo of this release is available here: http://andromeda.ayrus.net:8080/

Blog post for this release:
TBA

Planned for the next week:
N/A