Skip to main content

You are here

Data Conversion and Cleaning

What is data conversion?

Data conversion is the process of taking data stored in one database and moving it to another with minimal data loss. A successful data conversion is largely predicated upon getting the data out of your current solution. More, a successful data conversion requires that data are converted all at one time. Data submitted piecemeal is difficult, if not impossible, to convert. 

Since the structure of each database is different, no two conversions are alike. The conversion process may involve splitting the data in some fields apart, while combining the data in other fields. Data in multiple tables may be moved to a single table or data in single tables moved to multiple tables. For example, data conversion often involves the de-duplication of family information—taking the information from multiple children or records and creating a single-family record linked to each of the original child records. 

Data conversion does not fix any inherent data problems that are due to inconsistent data entry or system error—such as misspellings, duplicate records, missing information, etc. However, in all cases, the greatest care is always taken not to lose information during data conversion. Depending on your current system, we either convert your current system as is, or you will export your data from your current system and we will convert that data. During discovery we work with you to determine what is appropriate.

Do I have to send you all our data at one time?

Yes. Converting data is a one time event. Once you have provided us with your data, we spend multiple days manipulating and converting this data to ready it for import into your new solution. Should you discover at a later data that you have more data or you decide that you would like to include additional data—we cannot add that data to the already converted data. In most cases, we would need to start over and reconvert the entire data set from the beginning. This would delay your implementation and would add significant cost to your project. 

What is data cleaning?

Data cleaning is the fixing, standardizing and checking of the converted data once it is in your new solution. The process is multi-part and involves changing field values to match your iR solution’s needs, standardizing values in the same fields, removing duplicate records, adding missing data, and often verifying information with an outside source such as a school directory.

Some of these initial cleanup tasks are done by iR during conversion. Other steps require the assistance of school staff during the onsite implementation and beyond. In most all cases, data cleanup will continue for a period of time after your onsite visit is complete because there is not sufficient time during the implementation. Part of the implementation agenda is to help you determine what cleaning is essential for immediate needs and reporting, and then, to help you create a plan to complete the data cleanup over time.

How much data should be exported for data conversion?

Should we determine that the best plan is for you to export data from your current system, you should plan to export as much data as possible. Since the process is generally formulaic, converting 50,000 records doesn’t take much more time than converting 5,000. We cannot convert data in stages. Because the process often requires that multiple sources of data be combined, converting another few files later is difficult if not impossible and a second conversion will substantially increase your cost. Cleaning your data obviously may be longer with more records, but not so with conversion. We will work with you to help you determine exactly what to export.

Should we clean our data before exporting and sending it?

In general, the answer is no. We can often use advanced techniques to more efficiently locate and correct solution-wide data errors during the conversion process. Additionally, because you do not yet have a detailed understanding of how your new solution works and what it needs, it is better to let us decide how to deal with your data. The biggest problem overall in the conversion is if, while you are cleaning your data, you delete records or data that you don't think you need or you delete field information that you think has scrambled data. We cannot fix what never gets extracted. Also, trying to go record-by-record checking values in many systems is far too time consuming to be practical, which is partly how the data became unclean in the first place. Just export as much data as you possibly can and we will use advanced techniques to speed the cleaning process during conversion. Then, during the on-site implementation we’ll work with you to create a plan for completing any needed data cleaning.

How do we send you our data?

Whether you send us your complete solution or exported data files you will upload it to our secure server. We will send you explicit instructions for accessing our server and uploading your data as the date draws near.

What about running our old system with our new system and converting data again later?

It is nearly impossible to maintain two systems simultaneously and keep them both accurate. It is equally difficult to reconvert and merge data accurately at a later date. As stated above, a second conversion would essentially start the process over and increase the cost. Running both systems together for a while is generally a very poor plan. This triples the workload of the office—working in the old system, working in the new system and keeping both systems in sync. There will almost certainly be inconsistencies with double data entry that will be very time-consuming to sort out at a later date. Additionally, because every system stores data differently and generates reports differently, your staff will be challenged to do their work in the new system and will probably become confused and frustrated. Generally the value of the installation is lost and the school must repeat the entire orientation, paying for a complete new conversion and the on-site days. A school should plan to keep the old system available for reference to old data (perhaps with read only access) but should never attempt to run two systems simultaneously.