Big Data - Text Mining

Big data and text mining project in a validated environment

anic – a sought-after partner in validated environments

IDMP (Identification of Medicinal Products) is one of the big topics in the pharmaceutical industry at the moment. Manual maintenance and preparation of the data is often too expensive or is not manageable.

Anic was approached by a customer in early 2019 regarding a platform as an extension to a Hadoop data lake with greatly increased security requirements. Anic was asked to take over the creation of the concept and the implementation as a regulated system, since we already have many years of experience with GxP-regulated systems in the pharmaceutical sector. The solution was required to meet high availability (HA) requirements in order to remain accessible from the data lake.

IDMP Text Mining Project – Concept

Predefined applications had to be supplemented to meet the requirements. Nginx, Tomcat, Solr, MySQL / MariaDB, Elasticsearch and MongoDB were used. In addition to the system architecture, it was required that the load be estimated, the best possible integration into the existing infrastructure and the implementation of the security policies also be taken into account.

The initial design was ready in two weeks. Existing components such as the VMWare farm or the HTTP load balancer appliance had been integrated, all taking into account the customer’s qualification and QMS framework.


The technical implementation in the development environment, including documents for the approval of the platform, only took six weeks. Bespoke modules based on MongoDB and a custom high availability solution based on the SUSE HA extension for the text mining application were created.

After the data scientists had completed the pipelines to the data lake, the system went live in late 2019.

Anic once again proved with this that we are a strong partner for big data projects. We are familiar with the special requirements in the pharmaceutical sector and have made our contribution to the success of the customer with this installation in a validated and qualified environment.

What can we do for you? Take a look at a summary of our services under the following link: We can do this for you.

Get in touch with us, we would love to talk to you..

Related articles

Let’s Work together!

Let us help you envision the future of your company.

Please call me back.