1. A data quality analysis and management system comprising:
a processor;
an application service integration and communication interface to interface with internal and external systems to receive data;
a data quality testing module, executed by the processor, to perform data quality tests on the received data and to determine data quality statistics from the execution of the data quality tests, wherein the data quality tests include completeness, conformity, consistency, integrity and duplicity tests;
an error handler to execute remedial operations in response to data quality errors detected by the data quality testing module,
wherein the data quality testing module executes the completeness and the conformity tests in a first stage of data quality testing and the error handler performs data cleansing based at least on the conformity test, and
the data quality testing module executes the consistency, the integrity and the duplicity tests on the cleansed data in a second stage;
a data quality analysis and management engine to determine data quality cost metrics including cost of setup, cost of execution, internal data cost, and external data cost, and calculate a cost of data quality from the data quality cost metrics; and
a reporting module to generate a data quality scorecard including statistics determined from execution of the data quality tests by the data quality testing module and the cost of data quality determined by the data quality analysis and management engine.
2. The system of claim 1, wherein the cost of data quality calculated by the data quality analysis and management engine equals ((cost of setup+cost of execution)+(internal data cost+external data cost)), wherein the internal data cost equals (business remedy cost+information technology remedy cost), and the external data cost equals (the business remedy cost+the information technology remedy cost+external data support and communication cost).
3. The system of claim 2, wherein the cost of data quality is calculated as a function of a number of data quality errors detected by the data quality tests.
4. The system of claim 1, wherein the data quality scorecard, for different data sources or for different applications or for different categories of capabilities, indicates severity levels of data quality errors detected in response to the execution of the data quality tests, a total of successful records of the received data, a rejection percentage, a benchmark comparison for data quality error rate, and a trend indicator to indicate, for each of the data quality tests for which data quality errors were detected, whether the data quality errors are increasing or decreasing.
5. The system of claim 1, wherein the data quality analysis and management engine determines a forecast of data quality cost over time as a function of a cost increase factor per time period and severity of data errors detected by the data quality tests.
6. The system of claim 1, wherein the error handler determines different severity levels of data quality errors detected in response to the execution of the data quality tests, and different ones of the remedial operations are executed based on the different severity levels.
7. The system of claim 6, wherein the different severity levels comprise a fatal error, a hard error and a soft error, wherein error logging, rollback and reporting are the remedial operations for the fatal error, and rejecting a record in the received data, error logging, and reporting are the remedial operations for the hard error, and marking the record as an error, error logging, and reporting are the remedial operations for the soft error.
8. The system of claim 7, wherein a session of data processing on the received data is continued if a data quality error having a hard error or soft error is detected and the session is discontinued if a data quality error having a fatal error is detected.
9. The system of claim 1, in the first stage, the data quality testing module executes the completeness test on mandatory attributes in key fields in a record of the received data and executes the conformity test on the record,
wherein if a data quality error is detected in response to the execution of the completeness test on the mandatory attributes of the record or the conformity test on the record, the error handler performs at least one of the remedial operations related to a hard error severity level, and if no data quality error is detected, the quality testing module executes the completeness test on other attributes of the record.
10. The system of claim 9, wherein if a data quality error is detected in response to the execution of the completeness test on the other attributes of the record, the error handler performs at least one of the remedial operations related to a soft error severity level.
11. The system of claim 10, in the second stage, if the record has not been previously rejected based on the execution of the completeness and conformity tests, the data quality testing module executes the duplicity test on the record,
wherein if a data quality error is detected in response to the execution of the duplicity test, the error handler performs the at least one of the remedial operations related to the hard error severity level, and if no data quality error is detected, the data quality testing module executes the integrity test on the record.
12. The system of claim 11, wherein if a data quality error is detected in response to the execution of the integrity test on the record, the error handler performs the at least one of the remedial operation related to the soft error severity level on the record.
13. The system of claim 1, wherein in the first stage, the cleansed data is loaded into a staging table, and in the second stage, the data quality testing module executes the consistency, integrity and duplicity tests on the cleansed data in the staging table, and wherein the error handler performs at least one of the remedial operations on the data in the staging table in response to data quality errors detected in the second stage, and the data is subsequently loaded into a target table.
14. A non-transitory computer readable medium including machine readable instructions that are executed by at least one processor to:
receive data comprising a plurality of records from a data source via an application service integration and communication interface;
execute data quality tests on the received data and to determine data quality statistics from the execution of the data quality tests, wherein the data quality tests include completeness, conformity, consistency, integrity and duplicity tests, wherein the execution of the data quality tests includes executing the completeness and the conformity tests in a first stage and perform data cleansing based at least on the conformity test, and executing the consistency, the integrity and the duplicity tests on the cleansed data in a second stage and load the records;
determine data quality cost metrics including cost of setup, cost of execution, internal data cost, and external data cost;
calculate a cost of data quality from the data quality cost metrics; and
generate a data quality scorecard including statistics determined from execution of the data quality tests and the cost of data quality.
15. The non-transitory computer readable medium of claim 14, wherein the cost of data quality calculated by the data quality analysis and management engine equals ((cost of setup+cost of execution)+(internal data cost+external data cost)), wherein the internal data cost equals (business remedy cost+information technology remedy cost), and the external data cost equals (the business remedy cost+the information technology remedy cost+external data support and communication cost).
16. The non-transitory computer readable medium of claim 14, wherein the cost of data quality is calculated as a function of a number of data quality errors detected by the data quality tests.
17. The non-transitory computer readable medium of claim 14, wherein the machine readable instructions to execute the data quality tests in the first stage comprise instructions to:
execute the completeness test on mandatory attributes in key fields in a record of the records and execute the conformity test on the record,
wherein if a data quality error is detected in response to the execution of the completeness test on the mandatory attributes of the record or the conformity test on the record, perform at least one of the remedial operations related to a hard error severity level, and if no data quality error is detected, execute the completeness test on other attributes of the record, and
load the record to a staging table in a data repository.
18. The non-transitory computer readable medium of claim 14, wherein the machine readable instructions to execute the data quality tests in the second stage comprise instructions to:
execute the duplicity test on the record, wherein if a data quality error is detected in response to the execution of the duplicity test, perform at least one remedial operation related to the hard error severity level, and if no data quality error is detected, execute the integrity test on the record, and
if a data quality error is detected in response to the execution of the integrity test on the record, perform at least one remedial operation related to a soft error severity level on the record, and if a data quality error is not detected in response to the execution of the integrity test on the record, load the record into a target table in the data repository.
19. A method of performing data quality analysis and management comprising:
executing, by a processor, data quality tests on records received from a plurality data sources, wherein the data quality tests include completeness, conformity, consistency, integrity and duplicity tests, and wherein the execution of the data quality tests includes executing the completeness and the conformity tests in a first stage and performing data cleansing based at least on the conformity test, and executing the consistency, the integrity and the duplicity tests on the cleansed data in a second stage and loading the records in a database table;
determining data quality cost metrics including cost of setup, cost of execution, internal data cost, and external data cost;
calculating a cost of data quality from the data quality cost metrics, wherein the cost of data quality calculated by the data quality analysis and management engine equals ((cost of setup+cost of execution)+(internal data cost+external data cost)), wherein the internal data cost equals (business remedy cost+information technology remedy cost), and the external data cost equals (the business remedy cost+the information technology remedy cost+external data support and communication cost); and
generating a data quality scorecard including statistics determined from execution of the data quality tests and the cost of data quality.
20. The method of claim 19, comprising:
determining different severity levels of data quality errors detected in response to the execution of the data quality tests, and different remedial operations are executed based on the different severity levels.
The claims below are in addition to those above.
All refrences to claim(s) which appear below refer to the numbering after this setence.
1. Apparatus for measuring the efficiency of transport of modem relay packets over a packet network, comprising:
a first means for connecting to a first gateway of said modem relay connection;
a second means for connecting to a second gateway of said modem relay connection;
a computing device, operably connected to each of said first connecting means and said second connecting means through a network,
for running at least one terminal program for at least one of said gateways after a steady-state connection has been established between said first means and said second means;
for providing a reference modem data stream at a known modem stream generation rate to at least one of said gateways,
wherein said reference modem data stream has at least one digital file that is independent from setup and protocol files for said first and said second means; and
for receiving a transported modem data stream at a modem stream reception rate from a second one of said gateways after said reference modem data stream has passed through said gateways and said packet network; and
means for determining the transport efficiency of said packet network by comparison of said known modem stream generation rate of said reference modem data stream to a determined throughput rate of said received modem data stream.
2. The apparatus of claim 1, wherein:
said terminal program runs a Z-modem protocol for generating said modem data stream at a known generation rate.
3. The apparatus of claim 1, wherein said computing device:
runs a terminal program for each of said gateways;
provides a reference modem data stream at a known generation rate to each of said gateways;
receives a transported modem data stream from each of said gateways after said reference modem data stream has passed through said gateways and said packet network; and
determines the transport efficiency of said packet network by comparison of said known generation rate of said reference modem data stream to said determined throughput rate of said received modem data stream.
4. The apparatus of claim 1, wherein said throughput efficiency is measured after said gateways have negotiated appropriate protocols over said packet network.
5. The apparatus of claim 1, wherein said modem relay connection is established across a network.
6. The apparatus of claim 1, wherein said modem relay connection is established across a network simulator.
7. The apparatus of claim 6, further comprising:
means for control of network transport parameters of said network simulator.
8. The apparatus of claim 7, wherein said controlled parameters include packet loss and packet delay.
9. The apparatus of claim 8, wherein said parameters can be varied to selectively simulate uniform distribution of packet loss and random distribution of packet loss.
10. The apparatus of claim 7, further comprising:
means for providing alternative packet protocol algorithms to at least one of said gateways and measuring comparative throughput efficiency while maintaining fixed network conditions to provide a common evaluation reference.
11. The apparatus of claim 1, wherein:
said determination of said transport efficiency includes at least two iterative repetitions of said provision of said reference modem data stream and said reception of said transported modem data stream and said comparison of said known generation rate of said reference modem data stream to said reception rate of said received modem data stream; and
wherein said determination is based upon the average efficiency determined after a series of said iterations.
12. A method for measuring the efficiency of transport of modem relay packets over a packet network, comprising:
providing a reference modem data stream at a known generation rate to a first one of a plurality of gateways of a modem relay connection;
receiving a transported modem data stream from a second one of said gateways after said reference modem data stream has passed through said gateways and said packet network; and
after a rate negotiation and a steady-state have been reached between said gateways, comparing said known generation rate of said reference modem data stream to said determined throughput rate of said received modem data stream by fixing network conditions and varying one or more network conditions while transmitting a digital file as said reference modem data stream, to determine a network transport efficiency.
13. The method of claim 12, further comprising:
collecting a group of data representative of the network throughput efficiency under a number of network conditions and corresponding to a plurality of known file transfer protocols;
determining the network throughput efficiency values corresponding to the plurality of file transfer protocols and determining the average rate for a given protocol with given network conditions; and
generating a representation indicative of the relationship between modem relay system design and packet transport efficiency across said network.
14. A method to measure throughput efficiency of low speed modem relay over packet networks, comprising:
establishing a modem call over a modem relay system between a first modem to a second modem over a network through a modem relay system comprising a first gateway, connected locally to said first modem, and a second gateway, connected through a packet network to said first gateway and connected locally to said second modem;
transmitting a demodulated data stream of said modem call over said packet network;
establishing a steady-state connection over the packet network;
generating a plurality of reference modem data stream throughput rates of a first file type by repeatedly transmitting said first file type across from said first gateway to said second gateway as said reference modem data stream,
wherein said first file type is independent from setup and negotiation protocols of said modems and said gateways;
measuring a first modem data stream reception rate during said modem call by determining an average throughput rate of said first file type;
repeating said generating and said measuring for a second file type,
wherein said second file type is independent from setup and negotiation protocols of said modems and said gateways; and
measuring throughput efficiency of said modem relay by comparing said average modem data stream reception rates with said reference modem data stream generation rates.
15. The method of claim 14, wherein said measuring and said repeating are performed on both said first gateway and said second gateway and a comparison of said measuring from both said gateways are used to determine said throughput efficiency.