Good data management for efficient data handling

In modern companies, huge quantities of data records are created every day. Thanks to electronic data processing (EDP), collecting and organizing this information is no longer a problem. Customer data can be recorded in databases, and personnel data management can be easily automated. One simply imports the data onto the computer, and sophisticated algorithms take care of the rest.

However, increased networking and the growing volume of data have given rise to a range of new problems. For example, often multiple employees must be able to access the same database simultaneously. They have to be able to locate data at any time and should not need to reenter it multiple times. Above all, data must be protected from potential losses caused by hardware defects or mishandling, and shielded from hackers and other data thieves online. It’s also important to consider legal aspects such as mandatory retention periods or declarations of consent for the storage of personal data.

The complexity of this topic has led to the development of a new discipline in information technology: data management. This area is also the subject of much scientific research. Data science is still a relatively new branch of computer science that focuses on topics such as efficient storage and interconnection of data and effective searching of large databases.

Data management places certain demands on the handling of digital data. The term itself describes a process consisting of individual practices. Successful data management requires initial organization beginning at the point at which data is collected and entered. Data minimization and quality are two factors to consider here. While protecting content, you should ensure that data can be used effectively for the actual purpose for which it was collected (i.e. practicality should always be a priority). Lastly, an important part of data management involves determining which data needs to be archived and for how long. Unneeded data must be located quickly and deleted securely.

When planning how to handle data, start by asking yourself what types of data you’re dealing with. This is where it helps to divide data into categories so that you can proceed systematically without overlooking any areas:

• Personal data: Information directly related to specific persons; typical examples are names, phone numbers, and addresses. Other examples include measurement data and shopping behavior. Personal data can include customer data, data on a company’s own employees, or third parties. This data is subject to special protection.
• Sensitive company data: Every company should be particularly invested in carefully handling their internal data such as accounting data, tax documents, and company secrets. When it comes to data management, it’s important to define which information belongs to this category.
• Secondary data: Data that is acquired while collecting data for another purpose. One example might be data from video surveillance, which is usually installed to protect against burglary and theft. This data may also include license plates of customer vehicles. Another example is logs in the company network, which may store visitors’ IP addresses.
• Public data: Intentionally published and disseminated data; this includes information on a website and in company brochures. It’s important to comply with copyright regulations while also protecting your own data. In the US, your images are automatically protected under copyright law, whereas advertising slogans and company logos require registered trademarks.

The goal of data management is to efficiently integrate all processes from collection to storage or deletion of data. The process covers the full data “lifespan”. This gives rise to the term Data Lifecycle Management (DLM).

Data processing begins with data collection. The important factor here is data minimization, which means collecting only the minimum amount of information needed for a specific purpose. This concept plays a central role in the EU’s General Data Protection Regulation (GDPR). According to the law, data may only be processed if data subjects have provided their consent or if it is necessary for legal reasons (e.g. for drafting a contract). US organizations that handle any data about a European citizen are also required to comply with this law, and there are significant penalties for non-compliance.

Data quality can be ensured most effectively during data input. Careful data input eliminates the need for unnecessary subsequent queries and processing. Information should also be stored in the format in which it will be needed later. Any time data is transferred or converted, it can cause errors in the database.

The choice of data storage location and format is important. Data can be stored in a local folder or in the cloud. Each solution has its advantages and disadvantages. Locally stored files are easier to protect from unauthorized access. Cloud storage, on the other hand, is more scalable and fail-proof. It can make sense to use a combined solution for very important data.

Databases are the best option for storing large quantities of data. If you use special software (e.g. accounting or warehouse management software), you don’t have to consider the storage location. However, it’s important to ensure that the software is compatible with external systems. In addition, the software should allow you to create electronic backup files for submission to government agencies such as the Internal Revenue Service.

Data security is an important and complex topic in data management. Data must be protected from loss, unwanted changes, and unauthorized access. The Cybersecurity Framework published by the US National Institute of Standards and Technology (NIST) contains standards, guidelines, and practices to help organizations manage and mitigate cybersecurity risks. Similarly, ISO standard 27001 provides a set of best practices for identifying and addressing information security risks. Organizations that demonstrate compliance with this standard can achieve ISO 27001 certification.

Possible data security risks include:

• Hardware damage due to fire, water, or overvoltage
• Data loss due to mishandling
• Loss of data or systems rendered inoperative due to malware (encryption trojans, data theft)
• Data loss due to software errors
• Loss due to theft

Solutions for countering these risks include software-based protection mechanisms as well as organizational measures such as fire and intrusion detection systems.

You should keep the following principles in mind:

• Regular updates: It’s important to weigh up the options of automated and manual updates. The advantage of automated updates is that you won’t forget to perform them. An advantage of manual imports is that you can avoid updates that contain bugs.
• Secure passwords: Different strategies for password protection are available. It makes sense to ask employees to use complex passwords and change them regularly. However, if passwords are overly complex or changed too frequently, employees may fall into the habit of writing them down and storing them at their workplace.
• Backup strategy: One of the most important points is to use the right backup strategy. Important data should be backed fully and regularly using physically separate media. Data backups of databases present a special challenge. Under certain circumstances, you might not be able to copy open files while the system is running. Instead, you have to perform the backup from the application directly or use special software such as MySQLDump.
• Virus protection/firewall: Every IT system should provide up-to-date virus protection. Depending on the complexity of the network, your system should include a firewall and possibly an intrusion detection system.

Backups should be created automatically. Otherwise, there’s a significant risk that they will be omitted due to time pressure, convenience, or forgetfulness. Important data must be saved incrementally (i.e. in multiple versions). This means that only modified data records are backed up. If possible, older versions should be kept for a certain time in order to restore data in case it is accidentally deleted.

Ensuring the security of stored backups is an important topic in data management. Encryption trojans attempt to compromise any memory that can be accessed. In the worst case, backups that are saved on a permanently connected network or on external storage media will also be encrypted. This requires a system that prohibits access by ordinary users and only temporarily connects to the storage media during the backup.

It’s important to distinguish between data protection and data security, even though there is some overlap. The goal of data protection is to ensure that unauthorized persons cannot access confidential data. This involves preventing external access, which requires the installation of data security measures. It also entails using rights management technology to prevent internal access to personal data. With this software technology, access is denied for certain employees, or data records are only partially displayed depending on their privileges. Encrypted data transmission and storage can provide additional protection. This protects sensitive data in the event that the hardware is accessed by unauthorized employees or thieves in the event of a break-in.

Data management must be integrated into your company’s workflows as practically and intuitively as possible. This ensures maximum acceptance among employees and optimizes effectiveness. Many data management practices are useful for improving efficiency. Collecting unnecessary data takes time and may annoy customers. Orderly and secure data storage improves productivity.

It can therefore make sense to implement a data governance policy that defines how to handle data within your company. This means focusing on data quality and using software features such as auto-correct to improve data. It also involves defining standard formulations and terms.

Archiving data that is no longer immediately needed is another important task. Companies must do this when they have a legal obligation to retain data such as invoices and tax documents. Therefore, you should include archiving in your data management concept.

Separate storage can be advantageous. It reduces the volume of current data backups and ensures data protection. Not all storage media backup options are appropriate for archiving. For example, hard drives should be switched on regularly to ensure that they are working. Optical storage media such as CDs are sensitive to environmental conditions and have a limited lifespan. Tape drives with magnetic tapes are ideal. However, the disadvantage of these drives is that they are expensive and cumbersome. Their advantage is that the backup tapes themselves are inexpensive and durable.

Data that is no longer required should be deleted. That way, you’ll no longer be responsible for protecting it. Your data management concept should therefore ensure that such data can be selected and deleted separately. Above all, personal data must be deleted securely.

When data is deleted using operating system functions, it usually only means that the data is released for overwriting. The data actually continues to exist on the hard drive until the storage space is needed and it is then overwritten.

Unlike the EU, the US does not have an all-encompassing data privacy law. However, personal information is generally protected under the US Privacy Act as well as several other laws that apply to certain areas:

• Health Insurance Portability and Accountability Act (HIPAA)
• Fair Credit Reporting Act (FCRA)
• Children’s Online Privacy Protection Act (COPPA)
• Gramm-Leach-Bliley Act (GLBA)

Compliance with these laws is mandatory. Your organization and its representatives can be held directly liable for any non-compliance that results in data breaches or misuse of personal data. Therefore, it’s wise to appoint a privacy officer to ensure compliance with these regulations.

How data management is organized depends on the size of the company. Various approaches to integrated solutions are available on the market. Some are specifically dedicated to evaluating and optimizing the use of existing data (e.g. for advertising purposes). Others aim to improve productivity using all available data. Possible options include:

• Enterprise Resource Planning systems (ERP): These systems provide the most comprehensive approach. They record and take into account all company resources. This includes personnel, work equipment, and material data. Well-known commercial providers of ERP software include SAP, Sage, Oracle, and Microsoft. There are also free software solutions such as Odoo and OpenZ.
• Master data management: These systems are used for centralizing and revising a company’s core data. This includes employee data, customer data, and information about work equipment. The goal is to ensure uniform data quality, resulting in improved usability. This is the most common approach in ERP systems.
• Content management systems (CMS): These are predominantly information management systems (e.g. a central company intranet). Thanks to their great flexibility, they also allow for other aspects such as form management and database integration.
• Document management systems (DMS): A sub-area of data management; these systems provide forms and functions such as filing and archiving.

Data management is a dynamic process and must always be adapted to current needs. This generates new challenges.

Data volume is constantly increasing. As a result, there are high demands on the scalability of storage and backup capacity, on the organization and searchability of required data. The more data you collect, the more important data minimization becomes. Therefore, you should increasingly focus on extracting only the most important data.

Network administrators are constantly confronted with new dangers. Information theft via social engineering and sabotage using encryption trojans are just two possible scenarios. The more a company digitizes its data, the more dependent it becomes on the functionality of the system it uses. That’s why it’s important to stay up to date and informed on emerging risks, and take precautions to prevent hardware failure or loss of access to your own systems.

When the GDPR law was first introduced in Europe, it caused a lot of uncertainty and created a heavy workload for many companies. It also affected US companies, which are required to comply with the law if they hold data about European citizens. Data privacy is not highly regulated at the federal level in the United States. However, several states have drafted their own legislation to protect consumer data. For example, California’s Consumer Privacy Act gives consumers enhanced rights regarding how their personal data is handled. As further laws are introduced and existing regulations are changed, organizations will have to make adjustments that may also affect their data management.

Data management requires you to take into account any changes to your company’s structure or processes. You can plan ahead for such changes by using systems that are easy to expand or migrate. Regular employee training on in-house data governance policies requires additional effort.

Of course, data management can take up considerable time and distract a business from its core focus. However, when considering the individual practices involved, it soon becomes obvious that they’re necessary and useful. They are designed to help businesses comply with legal requirements, ensure greater security of their data, and improve the effectiveness of work processes. Therefore, your time will be well-invested.