Data Hierarchy and quality
Ok, this entry is a bit academic. Take it if it is useful for you.
If data is defined as unprocessed facts (as stated in here in Infogineering website: http://www.infogineering.net/data-information-knowledge.htm), then we don't have problem with its accuracy (but we might have other quality issues), because they are facts. Facts are indisputable. It is common that the term data and information are often used interchangeably.
I was once confused, whether to use the term 'data quality' or 'information quality' . Well, the understanding that I'm holding on is this. We can use both terms, depending on what we are dealing with.
Is it the unprocessed facts that we care about, or its processed version? Then, we can decide which term to use. So if we have data of a patient: "Temperature: 40'C, Date of diagnosis: 18-Sept-1978, Medication give: unknown", we can say it has completeness problem (one of data quality issues) as the medication data is missing. Then if the information processed from this data is : "The patient is healthy", then we have accuracy problem.
Here comes the term knowledge and wisdom that require one to distinguish them. The hierarchy describes the relationship among these terms. In data processing, it is important for us to understand the differences. Apparently, wisdom is the highest level of the hierarchy, that may separate humans with machines. Nevertheless, with machine learning advancement, people try to train machines (robots) to gain wisdom. After all, machines are now capable to (at least) produce knowledge.
(source: http://www.cvev.org/difference-between-data-information-and-knowledge.html)
So what is the purpose of data analytics, that becomes so popular now--in the big data era?
I agree that it is an advantage to have big data because we can gain information that can be used to harvest knowledge more than ever before. We can have bigger samples, that is useful to represent the whole population. This is what data analytics for. So it is left for human to use the knowledge 'wisely' (the wisdom level). I am saying this with the assumption that machine is not wise. If wisdom can be generated from machine, we need to make sure that the wisdom produced is as we expected. Otherwise, we might need to race (or fight) against the misbehaved machines, the machine that broke our trust. (This reminds me about I-Robot movie played by Will Smith--you can watch the trailer here)
But, analytic depends on how good the data is (its quality). If data is at its lowest level has quality issues, it will propagate to other levels. Thus it is so crucial to prepare data that is fit for its use as early as possible, that is free from any quality issues. Within data analytic life cycle, the data preparation is a stage where issues regarding data quality is detected and perhaps being resolved. With data that is of various formats and structures, this task is becoming more challenging due to the volumes of it.
I am keen to investigate what are the current practices for data quality assurance during data preparation stage. What are the methods, or tools used to handle the so called dirty data? Starting from detection, and how people react to it, if dirty data exists.
Until the next writing peeps. All the best!
If data is defined as unprocessed facts (as stated in here in Infogineering website: http://www.infogineering.net/data-information-knowledge.htm), then we don't have problem with its accuracy (but we might have other quality issues), because they are facts. Facts are indisputable. It is common that the term data and information are often used interchangeably.
I was once confused, whether to use the term 'data quality' or 'information quality' . Well, the understanding that I'm holding on is this. We can use both terms, depending on what we are dealing with.
Is it the unprocessed facts that we care about, or its processed version? Then, we can decide which term to use. So if we have data of a patient: "Temperature: 40'C, Date of diagnosis: 18-Sept-1978, Medication give: unknown", we can say it has completeness problem (one of data quality issues) as the medication data is missing. Then if the information processed from this data is : "The patient is healthy", then we have accuracy problem.
Here comes the term knowledge and wisdom that require one to distinguish them. The hierarchy describes the relationship among these terms. In data processing, it is important for us to understand the differences. Apparently, wisdom is the highest level of the hierarchy, that may separate humans with machines. Nevertheless, with machine learning advancement, people try to train machines (robots) to gain wisdom. After all, machines are now capable to (at least) produce knowledge.
So what is the purpose of data analytics, that becomes so popular now--in the big data era?
I agree that it is an advantage to have big data because we can gain information that can be used to harvest knowledge more than ever before. We can have bigger samples, that is useful to represent the whole population. This is what data analytics for. So it is left for human to use the knowledge 'wisely' (the wisdom level). I am saying this with the assumption that machine is not wise. If wisdom can be generated from machine, we need to make sure that the wisdom produced is as we expected. Otherwise, we might need to race (or fight) against the misbehaved machines, the machine that broke our trust. (This reminds me about I-Robot movie played by Will Smith--you can watch the trailer here)
But, analytic depends on how good the data is (its quality). If data is at its lowest level has quality issues, it will propagate to other levels. Thus it is so crucial to prepare data that is fit for its use as early as possible, that is free from any quality issues. Within data analytic life cycle, the data preparation is a stage where issues regarding data quality is detected and perhaps being resolved. With data that is of various formats and structures, this task is becoming more challenging due to the volumes of it.
I am keen to investigate what are the current practices for data quality assurance during data preparation stage. What are the methods, or tools used to handle the so called dirty data? Starting from detection, and how people react to it, if dirty data exists.
Until the next writing peeps. All the best!
Comments