Case Study - Dept. of Justice & Equality
Data & System Integration
Department of Justice & Equality
Case Study
How the Department of Justice & Equality solved its Lotus Notes/Domino problem once and for all.
Based on an Interview with Derek Coffey, Head of Development, Department of Justice & Equality – October 2019
Derek Coffey came to work in the Department of Justice & Equality in 2015 from the private sector on the back of the Toland Report.
“There is a serious lack of integrated and timely data.” (Toland Report)
This landmark report highlighted areas needing improvement within the Department of Justice & Equality. Derek and his team developed a general plan to address the issues in the report, data being a key aspect.
“in the Department’s Strategy Statement data is recognised as a key enabler.” (Implementation of Toland Report recommendations)
“A lot of work around how we could look at systems differently, an agile approach to development, better engagement with end users around the development process and in the collaboration space.” (Derek Coffey)
This overall departmental strategy gave rise to a multitude of challenges, a key one was legacy data, it’s migration and integration. The question they faced was “How you might migrate away from Lotus Notes”.The Challenge:
One of the challenges facing Derek and his team was the Lotus Notes legacy data. How to get it, how to integrate it and how to use it.
“Lotus Notes….. was not sustainable going forward.”
“So one of our significant challenges is we have all this Lotus Notes Data, it is a flat file structure, it is difficult to get anything to work well with it.”
The Dept. needed a solution
“which would serve up the data the way we needed it.”
Lotus Notes ”created a problem”.
“We were trying to establish that we were going to be Microsoft SQL based as a department. Part of my role is delivering quality data to our research and analytics team. So we’d determined that we’d do that through Microsoft SQL as our common database platform.”
“That was one of the challenges, how we comprehensively get data out of Lotus Notes and into Microsoft SQL in a way that was repeatable and resilient”.
“EastPoint had identified a platform called Darwino and yeah it does exactly what we needed it to do.”
“It allows us to use the inbuilt Lotus Notes replication to replicate the data out in a JSON format into Microsoft SQL, and then our data team can easily work with it from there. And we can consider how as we develop other applications in the future how we’d use Darwino as a data migration platform for those systems.”
EastPoint using Darwino solved the Lotus Notes problem in the Department of Justice & Equality.
Proof of Concept:
EastPoint were asked to set up a proof of concept.
“It went very well. The system was implemented quite quickly. We did a proof of concept in terms of a smaller data set. We had a challenge with an international terrorist incident. One of the terrorists was identified with an Irish immigration card in his pocket. We have a significant number of databases in immigration associated with the immigration process, and a lot of different Lotus Notes databases. And there was no easy way of carrying out searches across those, relied on a lot of people accessing different systems. So out of that we had a project to develop search product using elastic search, which we had already begun to evaluate. But rather than trying to point it at Lotus Notes databases, we used Darwino to replicate the data, and keep it in sync, practically real time over to Microsoft SQL, then indexed in Microsoft SQL as a search.“
The proof of concept was a success.
Troubleshooting:
Dealing with legacy data is not straightforward, but when issues arose EastPoint were there to help.
“But we did have some challenges, not with the application, but with the data. Some of those databases obviously went through a lot of changes over time, some fields got added, some became deprecated and neglected. There was some data quality issues that we could see in the replication. We had to work with the Darwino folk and EastPoint to kind of go through those and identify, ok is this an issue in Lotus Notes, in the data that is in Lotus Notes and the replication, or with the way Darwino is processing it. And in fairness EastPoint had significant expertise in both Darwino and Lotus Notes. They were able to troubleshoot those and get us answers pretty quick.”
Result:
“The search links you back to the original database, the fact that it is SQL allows us to index it easily for search purposes. So we were able to wrap up and get a significant number of databases, there were about fifteen of them that were a matter of interest, and just one after another we were able to put them through Darwino.”
Real time all the time. One of the strengths of Darwino is that it retrieves the data in real time, syncs the data in real time.
“The Darwino piece of it, is kind of set and forget as far as we are concerned.”
That is how well it works.