911maps is now open source!

I am open-sourcing the codebase for 911maps, the web app I have been building since early last year.

911maps is my vision for creating a framework that allows the public to map and analyze emergency incidents and responses, using data obtained from public safety authorities.

The inspiration

In 2018 I saw a few tweets posted by a local police chief from Ocean County that included a heatmap generated by his department’s computer aided dispatch (CAD) system about trends in his jurisdiction’s calls for service - and whether the data supported conventional wisdoms held by certain residents within the community.

The proprietary nature of the software provided by CAD vendors means that these advanced heatmapping capabilities are only available to law enforcement agencies and not the public.

Seeing these visualizations in my Twitter feed got me thinking:

Why should the public, researchers and other concerned stakeholders not be able to produce these same visualizations and see what is taking place in their communities?

Data should objectively tell the story about trends in emergency incidents in cities and municipalities. Now, not all data is truthful or useful for this purpose, but we should at least be able to assume that the records created by public safety authorities will accurately reflect trends in calls for service, making it suitable for research and analysis purposes.

Active911

I was able to execute on my idea for creating my own heatmapping system when I learned of agencies in my community adopting Active911 to coordinate their responses to emergency incidents using their smartphones. Active911 is a mobile app with a web component that allows first responders to coordinate their responses to incidents via a mobile app. It does so by receiving data from the CAD utilized by dispatch centers.

Multiple endpoints are provided by Active911’s backend, including RSS feeds and other mechanisms for sharing data. Some agencies that have adopted the system provide open access to the real-time RSS feeds of call data, and even have them embedded as a widget on their websites, but I have not been able to find any in New Jersey that do so.

Also of note, the Active911 system takes the address information entered by dispatchers and geocodes it using the Google Maps geocoding API, storing the resulting latitude and longitude of the call as columns in the database. This makes the dataset useful for using GIS tools and libraries to study trends in public safety data.

Obtaining the data

Since there were no agencies providing real-time access to their feeds, the data has to be obtained after-the-fact. Building upon my previous work with public records, the data stored in Active911 can be requested from governmental dispatch agencies under the state’s Open Public Records Act (OPRA). The columns of data can easily be exported in a few clicks from the web interface available to dispatchers and system administrators, resulting in a CSV file suitable for analysis.

The only caveat to be aware of with this approach is that Active911 only retains call records for the past 3 months. If you want to go further back than that you could run into issues. For my purposes, I found that requesting the data on a monthly basis was the best way to obtain consistent data, while being considerate of agency resources and my own time dealing with the back-and-forth of the public records request process.

On average, the monthly data exports I have been requesting that cover Ocean County have about 5 to 6000 incidents, of nearly every type on a countywide basis. This makes the data suitable for exploring any number of incidents, such as fires, car accidents and medical calls.

Example heatmap: 3 months worth of fire & EMS call activity in my hometown, Lacey Township:

Why build a custom platform?

911maps is my approach to building a platform to utilize this data for researching trends in public safety incidents. As I have catalogued the development of my platform, some have suggested that I use existing solutions, such as ArcGIS and other proprietary platforms. While these platforms certainly have their place, I wanted to use open-source technologies that were readily available and customizable to suit my vision without the pricetag of proprietary technologies.

Additionally, another longterm goal that necessitates building a custom platform is my plan of adapting this system to work with other dispatch systems. For example, the New Brunswick Police Department in Middlesex County can provide so-called “dispatch logs” in PDF format, giving just a brief synopsis of the call, time and location. One can take the data from the PDF file and convert it into a CSV by running it through a program like Tabula and subsequently geocode the addresses to obtain the latitude and longitude pairs necessary for properly mapping the data.

Putting it all together

The only real cleaning that is needed is the timestamps of the calls. Using a simple sed expression on the Linux commandline we can get them into a format that will play nice with Django’s DateTimeField subclass. Once the data being in the format needed by Django, I added the ability to upload data directly from the administrative interface of the web app using django-import-export.

I’ve learned quite a bit from building the web app, from the finer points of defining database models in Django, to cleaning the CSV datasets to make them suitable for importing, and passing variables from Django to the embedded JavaScript code that handles the heatmapping and the trial-and-error process needed to get the codebase to a working state.

The mapping is handled by the excellent Leaflet.js library, which has extra plugins for heatmaps. Using 911maps it is possible to create heatmaps for nearly every type of call naturem, or even by specific municipality or dispatch agency, enabling new levels of insight into public safety issues.

Future plans

I am happy with the progress that has been made thus far with 911maps. It took nearly a year to get the project to this point. As the codebase has matured, I have explored research collaborations with the faculty of the Addiction Studies Department at Ocean County College, which will be continuing as I begin working.

I have also obtained a new statewide dataset from another law enforcement agency that is showing great promise as source for understanding the extent of the opiod crisis in New Jersey, which I will be extending 911maps to cover.

As much as I have made progress with the codebase and learned about new visualization and analysis techniques in graduate school, I have come to the conclusion that I have incurred some technical debts with the current version of the code. Therefore, I have come to the conclusion that 911maps needs to be written from the ground up with a new approach in order to make it more maintainable in the long term and useful for analysis and research purposes.

So at this point, I am releasing the code as it exists now as open-source software under the GNU GPLv3 license. This iteration is going to stay as-is for now, and when it is rewritten that version will supersede this version and be released in a new repository. You can download the code here from Github here. I also have a development server of 911maps running with 3 months of sample data at 911ma.ps. In the spirit full disclosure, some parts of 911maps are broken or could be improved, but right now I am choosing to focus on rewriting it from the ground up rather than addressing these issues as it is a more efficient use of my resources.

In order to deploy 911maps you should have some understanding of the Django web framework and Python. I will not rehash the basics of those in this post. The install-dev.sh script in the scripts folder of the repository maybe helpful for getting a copy going.

If you would like to support my development of open-source data science platforms such as this, please consider supporting me on Patreon

Disclaimer: This project is neither endorsed nor affiliated with Active911, Inc. or any law enforcement agency and no claim is made to their copyrighted technologies. All data used in this project was lawfully obtained under the New Jersey Open Public Records Act.

ABSOLUTELY NO WARRANTY, express or implied or technical support will be provided for the open-source code released by the author and it is used at your own risk.