Enhancement and automation of the file exchange platform

The system functions as a file exchange platform, where documents are processed on their server, allowing their clients to receive new files with required info. Meanwhile, the original documents are maintained in paper form, electronic form in pdf or EDI text file. To ensure proper processing, these physical documents must be converted into the appropriate digital format for further usage. 

A key aspect of the system includes handling documents of a certain standard. These specific forms need to be accurately converted and integrated for proper workflow automation. Another challenge is the system's accumulation of large volumes of data over 13 years. Much of this data has not been accessed for an extended period and, in some cases, is even duplicated, leading to inefficiencies in storage and data management. 

Our primary focus was on archiving existing data and optimizing data storage. We implemented a distributed database solution, organizing the data efficiently by storing 10 years of historical data in one location, while keeping the most recent three years of active data in a separate, easily accessible repository. In addition, unnecessary data has been deleted. 

To further enhance the system’s performance, we redesigned the Extract, Transform, and Load (ETL) processes. This involved streamlining and optimizing the data flow, ensuring that files are efficiently converted into the correct storage format and easily accessed for analysis. We also automated several tasks using Directed Acyclic Graphs (DAGs). 

It is worth mentioning that the client does not receive raw data but is handled by the external processor. The task was to double-check their version and whether it was uploaded correctly to the client’s server. So that there is no mixing of information from different pharmacies and patients. 

As a result, all tasks were successfully completed, leading to significant improvements in efficiency. Through the implementation of data archiving, the amount of storage space consumed has been reduced by 80%, optimizing overall system performance. Additionally, the introduction of automated data archiving lays the foundation for future scalability, ensuring that data management can be handled seamlessly and more efficiently moving forward.