US20110010398A1 - System and Method for Organizing Data - Google Patents

System and Method for Organizing Data Download PDF

Info

Publication number
US20110010398A1
US20110010398A1 US12/758,180 US75818010A US2011010398A1 US 20110010398 A1 US20110010398 A1 US 20110010398A1 US 75818010 A US75818010 A US 75818010A US 2011010398 A1 US2011010398 A1 US 2011010398A1
Authority
US
United States
Prior art keywords
vector
data
database
distilled
records
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/758,180
Inventor
Bjorn J. Gruenwald
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to US12/758,180 priority Critical patent/US20110010398A1/en
Publication of US20110010398A1 publication Critical patent/US20110010398A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33Querying
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/258Data format conversion from or to a database
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2365Ensuring data consistency and integrity
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99937Sorting
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99942Manipulating data structure, e.g. compression, compaction, compilation
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users

Definitions

  • the present invention relates to database systems and more particularly, to a system and method for organizing data in a database system.
  • database systems are designed to organize, store and retrieve data in such a way that the data in the database is useful. For example, the data, or subsets of the data, may be searched, sorted, organized and/or combined with other data.
  • the usefulness of a particular database system is dependent on the integrity (i.e., the accuracy and/or correctness) of the data in the database system.
  • Data integrity is affected by the degree of “disorder” in the data stored. Disorder may occur in the form of erroneous or incomplete data such as duplicate data, fragmented data, false data, etc.
  • existing data may be edited and processed, and as a result, additional errors may be introduced.
  • new data may be introduced.
  • data conversion may be required or additional fields may become necessary.
  • the data in the database may simply become outdated over time.
  • the present invention provides a system and method for organizing data in a database system.
  • the present invention derives a distilled database of accurate data from raw data included in one or more raw data sources.
  • the raw data is converted from its original format(s) to a numeric format.
  • the raw data is represented as a vector having numeric elements.
  • various mathematical operations such as correlation functions, pattern recognition methods, or other similar numeric methods, may be performed on these vectors to determine how content in a particular vector corresponds to others vectors in a “distilled” or reference database.
  • the distilled database is formed from sets of one or more related vectors that are believed to be unique (e.g., orthogonal) with respect to the other sets.
  • new data may be screened to ensure that new errors are not introduced into the distilled database.
  • the new data may be also evaluated to determine whether it is unique or whether it includes better information than that already present in the distilled database.
  • the new data is added to the distilled database accordingly.
  • raw data is converted into a numeric format based on a number system having an appropriate radix.
  • An appropriate radix is determined according to the type of information included in the raw data. For example, for raw data generally comprised of alpha-numeric characters, an appropriate radix may be greater than or equal to the number of different alpha-numeric characters present in the raw data. Using such a number system allows raw data to be represented numerically, allowing for manipulation through various well-known mathematical operations.
  • the number system may be selected so that the numbers themselves retain semantic significance to the raw data they represent.
  • the numerals in the number system are selected so that they correspond to the raw data.
  • the numerals are selected to correspond to the alphanumeric characters they represent.
  • Another feature of the present invention is that once the raw data is represented as vectors in an appropriate number system, the represented data may be efficiently manipulated in the database (e.g., sorted, etc.) using various well-known techniques. Furthermore, various well-known mathematical operations may be performed on the vectors to analyze the data content. These mathematical operations may include correlation functions, eigenvector analyses, pattern recognition methods, and others as would be apparent.
  • Still another feature of the present invention is that the raw data is incorporated into a distilled database.
  • the distilled database represents the best information extracted from the raw data without having any data disorder.
  • Yet another feature of the present invention is that new data may be compared to the distilled database to determine whether the new data actually includes any new information or content not already present in the distilled database. Any new information not already in the distilled database is added to the distilled database without adding any disorder. In this manner, the integrity of the distilled database may be maintained.
  • FIG. 1 illustrates a processing system in which the present invention may be implemented.
  • FIG. 2 illustrates stages of data processed by one embodiment of the present invention.
  • FIG. 3 is a flow diagram for converting raw data from its original format into a numeric format in accordance with one embodiment of the present invention.
  • FIG. 4 illustrates a data record suitable for use with the present invention.
  • FIG. 5 illustrates raw data tables suitable for use with the present invention.
  • FIG. 6 illustrates reference data tables, representing data formatted in accordance with an embodiment of the present invention.
  • FIG. 7 is a flow diagram for analyzing reference data in accordance with an embodiment of the present invention.
  • FIG. 8 illustrates distilled data table, representing related data correlated in accordance with an embodiment of the present invention.
  • FIG. 9 illustrates an example of data clustering in a two-dimensional space.
  • the present invention is directed to a system and method for organizing data in a database system.
  • the present invention is described below with respect to various exemplary embodiments, particularly with respect to various database applications. However, various features of the present invention may be extended to other areas as would be apparent.
  • the present invention may be applicable to many database applications where large amounts of seemingly unrelated data must be compiled, stored, manipulated, and/or analyzed to determine the various relationships present in the content represented by the data. More particularly, the present invention provides a method for achieving and maintaining the integrity (i.e., accuracy and correctness) of data in a database system, even when that data initially possesses a high degree of disorder.
  • disorder refers to data that is duplicative, erroneous, incomplete, imprecise, false or otherwise incorrect or redundant. Disorder may present itself in the database system in many ways as would be apparent.
  • One embodiment of the present invention is used to maintain a database associated with accounts receivable.
  • a company may collect data relating to various persons, businesses and/or accounts from one or more sources. These sources may include, for example, credit card companies, financial institutions, banks, retail, and wholesale businesses and other such sources. While each of these sources may provide data relating to various accounts, each source may provide data representing different information based on its own needs. Furthermore, this data may be organized in entirely different ways. For example, a wholesale distributor may have data corresponding to accounts receivable corresponding to business accounts. Such data may be organized by account numbers, with each data record having data fields identifying an account number, a business associated with that account number, an address of that business, and an amount owed on the account.
  • a retail company may have data records representing similar information but based on accounts corresponding to individuals as well as businesses.
  • other types of sources may provide different types of data.
  • the scientific institutions may provide scientific data with respect to various areas of research.
  • Industrial companies may provide industrial data with respect to raw materials, manufacturing, production, and/or supply.
  • Courts or other types of legal institutions may provide legal data with respect to legal status, judgments, bankruptcy, and/or liens.
  • the present invention may use data from a wide variety of sources.
  • a database may be maintained to implement an integrated billing and order control system.
  • this embodiment may include data records corresponding to inventory, data records corresponding to suppliers of the inventory, and data records corresponding to purchasers of the inventory.
  • Inventory data may be organized by part numbers, with each data record having data fields identifying an internal part number, an external part number (i.e., supplier part number), a quantity on hand, a quantity expected to ship, a quantity expected to be received, a wholesale price, and a retail price.
  • Supplier data may be organized by a supplier number; and customer data may be organized by a customer number.
  • Data records corresponding to each of these records may include data fields identifying a part number, a part price, a quantity ordered, a ship data, and other such information.
  • Another embodiment of the present invention may include an enterprise storage system that consolidates corporate information from multiple, dissimilar sources and makes that information available to users on the corporate network regardless of the type of the data, the type of computer that generated the data, or the type of computer that requested the data.
  • Still another embodiment of the present invention includes a business intelligence system that warehouses and markets information and allows that information to be processed and analyzed on-line.
  • the present invention enables raw data collected from different sources to be analyzed and distilled into a collection of accurate data, organized in a way that is useful for a particular application.
  • the present invention may produce a distilled database in which related data, such as data relating to a particular supplier or customer, may be identified as such.
  • related data such as data relating to a particular supplier or customer
  • duplicate data corresponding to the same supplier or customer may be identified and/or discarded, and erroneous data associated with the supplier or customer may be identified, analyzed, and possibly corrected.
  • the present invention may be implemented in hardware or software, or a combination of both.
  • the present invention is implemented as a software program executing in a programmable processing system including a processor, a data storage system, and input and output devices.
  • a programmable processing system including a processor, a data storage system, and input and output devices.
  • FIG. 1 An example of such a system 100 is illustrated in FIG. 1 .
  • System 100 may include a processor 110 , a memory 120 , a storage device 130 , and an I/O controller 140 , coupled to one another by a processor bus 150 .
  • I/O controller 140 is also coupled via an 110 bus 160 to various input and output devices, such as a keyboard 170 , a mouse 180 , and a display 190 .
  • Other components may be included in the system 100 as would be apparent.
  • FIG. 2 illustrates various forms of data processed by the present invention.
  • Raw data 210 may be collected from one or more sources, such as raw data 210 A and raw data 210 B.
  • raw data simply refers to data as it is received from a particular source. Additional sources of raw data 210 may be included as would be apparent.
  • raw data 210 from various sources is converted into a numerical format and stored in a reference database 220 .
  • data dialysis the present invention “purifies” raw data 210 to form reference data in reference database 220 .
  • Reference database 220 includes all the information found in raw data 210 including duplicate, incomplete, inconsistent, and erroneous data.
  • Distilled data stored in a distilled database 230 is derived from the reference data of reference database 220 .
  • Distilled data represents the “accurate” data available from raw data 210 .
  • Distilled database 230 includes the unique information found in raw data 210 . Distilled data thus represents the best information available from raw data 210 .
  • the present invention further provides for using distilled database 230 to analyze and verify new data 240 , which may also be used to update the reference database 220 and distilled database 230 as appropriate.
  • raw data 210 is a collection of data collected from various sources, such as order processing, shipping, receiving, accounts payable and accounts receivable, etc.
  • This raw data 210 may include data records that are related but have different data fields, duplicate data records, data records having one or more erroneous data fields, etc.
  • the present invention converts raw data 210 from their original formats and data structures (which may vary based on the source) into a numeric format and stores this reference data in reference database 220 .
  • the reference data is then compared and analyzed to distill the best information available.
  • this best information may be stored as distilled data in distilled database 230 . This process is now described.
  • FIG. 3 illustrates the process by which raw data 210 is converted into reference data in reference database 220 according to one embodiment of the present invention.
  • raw data 210 is collected from a raw data source.
  • raw data 210 may include data from one or more sources such as raw data 210 A and raw 210 B.
  • data refers to the physical digital representation of information
  • data “content” refers to the meaning of, or information included in or represented by that data.
  • the different records in raw data 210 may include similar types of data content. For example, in a billing context, different records in raw data 210 may all include data content relating to a particular account.
  • Raw data 210 will typically be received in the form of data records 400 , as illustrated in FIG. 4 .
  • Each data record 400 generally includes related information, such as information for a specific individual, company, or account.
  • Each data record 400 stores this information in one or more data fields 410 .
  • Examples of possible data fields 410 include, for example, an account number, a last name, a first name, a company name, an account balance, etc.
  • Each data field 410 may include one or more data elements 420 for representing information for that specific record and specific field.
  • Data elements 420 may exist in various formats, such as alphanumeric, numeric, ASCII, and EBCDIC, or other representation as would be apparent.
  • Raw data 210 collected from different sources may be formatted differently.
  • Data records 400 may include different data fields 410 , and the information included in data fields 410 may be represented using data elements 420 in different formats, as would also be apparent.
  • Examples of raw data 210 are illustrated in raw data tables 510 , 520 , and 530 of FIG. 5 .
  • Data records such as data record 510 - 1 and data record 510 - 2 , are illustrated as rows of raw data tables 510 , 520 , and 530
  • data fields such as data field 510 -A and data field 510 -B, are illustrated as columns of raw data tables 510 , 520 , and 530 .
  • the tables illustrated in FIG. 5 are examples of data that might be found in various embodiments of the present invention. In other embodiments, data may come from many sources and may be formatted as databases having a much larger number of data records and/or data fields, as would be apparent.
  • the present invention converts raw data 210 from its original representation (which may be in alphanumeric, numeric, ASCII, EBCDIC, or other similar formats) to a numeric representation. This ensures that reference data is represented in the same manner. Thus, the reference data, including that data from different sources, may be similarly processed.
  • raw data 210 is converted from its original representation into an appropriate numeric representation.
  • An appropriate numeric representation uses a number system in which each possible value of data element 420 may be represented by a unique digit or value in the number system.
  • a radix for the number system is selected such that the radix is at least as great as the number of possible values for a particular data element.
  • each data element may be one of only four values: A, G, C, and T.
  • a radix of four for the number system may be sufficient to represent each data element as a unique number.
  • One such number system may include the numbers A, G, C, and T.
  • such as number system may include the numbers A, G, C, T, and ⁇ , where ⁇ is the empty field value.
  • data elements 420 in raw data 210 are comprised of characters such as alphanumeric characters.
  • a radix of 40 is selected to represent the alphanumeric characters as illustrated in the table below. (Note that a minimum radix of 36 is required.) This radix is selected to accommodate the ten numeric characters “0”-“9” and the twenty-six alphabetic characters “A” to “Z” as well as to allow for several additional characters. In this embodiment, uppercase and lowercase characters are not distinguished from one another.
  • the base-40 number system includes the numbers 0-9, followed by A-Z, further followed by four additional numbers.
  • One of these numbers may used to represent an empty field. This number is used to represent a data field 410 that is empty or has no value (in contrast to a zero value).
  • Other numbers may be used, for example, to represent other types of information such as spaces or used as control information.
  • raw data 210 may be represented in a numeric fashion, facilitating straightforward mathematical manipulation.
  • Another benefit is that proper selection of both the radix and the numerals in the number system allows the represented content to maintain semantic significance, facilitating recognition the content of raw data 210 in its representation in the numeric format.
  • the word “JOHN” represented by the four alphanumeric characters “J” “O” “H” “N” may be represented in various number systems.
  • One such number system is a base-40 number system.
  • the selection of a radix and its corresponding number system may depend upon the number of bits used by processor 110 .
  • the number of bits used by processor 110 and the radix chosen for the number system define the number characters that can be represented by a data word in processor 110 . This relationship is governed according to the following equation:
  • N B *ln(2)/ln( R ),
  • N is the number of whole characters (i.e., fractional characters are discarded) represented by a data word of processor 110
  • B is the number of bits per data word
  • R is the selected radix.
  • the radix in addition to having a radix sufficiently large to maintain semantic significance, the radix may also be selected to maximize the number of characters represented by a single data word.
  • an appropriate radix may range from 36 to 40. This range maintains semantic significance while maximizing the number of characters represented by the 32-bit data word.
  • Other types of raw data and other sizes of data word may dictate other appropriate radix ranges in other embodiments of the present invention.
  • each data field 410 also dictates the precision required by the number as represented in processor 110 .
  • each data field 410 may only be six characters or data elements 420 wide for single precision operations in a 32-bit machine. In some embodiments of the present invention, this may be insufficient. In these embodiments, double, triple, or even quadruple precision may be required to represent the entire data field 410 as a single value. Double precision numbers are sufficient for up to twelve character data fields 410 ; triple precision numbers are sufficient for up to eighteen characters; and quadruple precision numbers are sufficient for up to twenty-four characters.
  • Alternate embodiments of the present invention may accommodate large data fields by breaking a large data field into one or more smaller data fields.
  • the large data fields may be broken at boundaries defined by spaces. For example, a data field representing an address such as “123 West Main Street” may be broken into four smaller data fields: ‘123’, ‘West’, ‘Main’, and ‘Street’.
  • the large data fields may also be broken at data word boundaries.
  • the smaller data fields might be: ‘123We’, ‘st ⁇ Mai’, ‘n ⁇ Stre’, and ‘et’, where the number ‘ ⁇ ’ is used to represent a space.
  • Other embodiments of the present invention may accommodate large data fields in other manners as would be apparent.
  • raw data 210 represented as a number is stored in a predefined data structure.
  • this data structure is a single-field table as illustrated by Tables 610 - 670 of FIG. 6 .
  • This data structure may vary.
  • the data structure may be a multiple-field table instead of a single-field table.
  • the data structures may be implemented with standard features such as table headers and indices, and as explained in greater detail below, may also include probability values for each record. These probability values represent the likelihood that the data in that record is complete. Higher probability values may indicate a higher probability of completeness, and lower probability values similarly may indicate a lower probability of completeness. This is described in further detail below. Initially, the probability values are set to 0.
  • Other embodiments may also include key numbers or identification numbers to aid in sorting and in maintaining relationships among the data records.
  • raw data 210 illustrated in FIG. 5 includes three tables 510 , 520 , and 530 .
  • Table 510 may represent raw data 210 from, for example, a company's accounts receivable system. Columns of table 510 represent data fields for an account number, a last name, a first initial, and additional fields for listing various orders processed for a particular individual. Rows of table 510 (such as 510 - 1 and 510 - 2 ) represent data records for different individuals.
  • Tables 520 and 530 may represent raw data 210 maintained by credit card companies. Columns of tables 520 and 530 represent data fields for an account number, a last name, a first name, and an address. Rows of tables 520 and 530 represent data records for specific accounts.
  • step 330 converts raw data 210 from the format illustrated in FIG. 5 into a format illustrated in FIG. 6 .
  • FIG. 6 illustrates raw data 210 , combined from the various raw data tables 510 , 520 , 530 of FIG. 5 , represented as numbers in a base-40 number system, and formatted as new tables (tables 610 - 670 ), which together may comprise reference database 220 .
  • Each reference database table 610 - 670 corresponds to an individual field from raw data tables 510 , 520 , and 530 of FIG. 5 . More specifically, data records of reference data tables 610 - 670 correspond to the data records of raw data table 510 , followed by the data records of raw data table 520 , followed by the data records of raw data table 530 .
  • a raw data table record has no information for a particular data field 410 represented in a reference table 610 - 670
  • a empty field value is entered in that field in the reference table.
  • the first data record 510 - 1 of Table 510 has no information about an address, and thus an empty field value is placed in the first position of table 670 .
  • Data is preferably stored in reference database 220 in such a way that all data corresponding to a single data record in a raw data table is readily identified.
  • data corresponding to any specific data record of the raw data tables (tables 510 , 520 , 530 ) is preferably represented in reference tables 610 - 670 as a “vector” of numeric data stored at an index i across reference tables 610 - 670 .
  • data corresponding to the sixth record 520 - 6 of raw data table 520 is represented in reference database tables 610 - 670 as a vector having coefficients formed from the tenth records 610 - 10 , 620 - 10 , 630 - 10 , 640 - 10 , 650 - 10 , 660 - 10 , and 670 - 10 of the tables 610 - 670 .
  • reference database 220 includes a new table 610 that does not correspond to any data field 410 in raw data 210 illustrated in FIG. 5 .
  • This table is a “key table” that identifies the related data in these data vectors.
  • reference database 220 comprised of the tables illustrated in FIG. 6 may include additional key tables for data fields. These may include a personal identification number (“P IDN”), an account identification number (“AIDN”), or other types of identification numbers. These key tables or identification numbers may be used to identify sets of related data vectors in reference database 220 .
  • key table 610 has a single field “PIDN,” which stands for personal identification number.
  • PIDN personal identification number.
  • Key table 610 provides a unique identifier such that a specific PIDN number never refers to more than one person represented in raw data 210 .
  • the PIDN number reflects the fact that many multiple records in raw data 210 may refer to the same person.
  • each data record in the key table 610 initially corresponds to a different data record represented in the raw data tables 510 , 520 , and 530 .
  • data record 610 - 10 in the key table 610 is implemented such that it includes identifiers (such as pointers or indices) for corresponding data in reference tables 620 - 670 , which together corresponds to a single record 520 - 6 in raw data table 520 .
  • a single individual may correspond to multiple PIDNs.
  • vector 4 defined by PIDN 4
  • vector 9 defined by PIDN 9
  • this person is initially assigned to two PIDN numbers—PIDN 4 and PIDN 9.
  • the present invention enables a determination whether PIDN 4 and PIDN 9 do, in fact, refer to the same individual, and if so, assigns a single PIDN to this individual.
  • some embodiments may assign a new PIDN number to individuals so determined and a reference to the old PIDN number may be retained.
  • records are represented in the reference database tables 610 - 670 as vectors having coefficients of base-40 numbers across eight one-field tables.
  • This numeric representation allows the data to be analyzed using straightforward mathematical operations that may be used to, for example, produce correlations, calculate eigenvectors, perform various coordinate transformations, and utilize various pattern recognition analyses. These operations may, in turn, be used to provide or derive information about the records and their relationships to one another. By using small, one-field tables, these operations may be performed quickly.
  • representation in base-40 numbers with raw data 210 including alphanumeric characters allows content of raw data 210 to retain its semantic significance.
  • a data dialysis process 700 is applied to distill the most accurate data for inclusion in distilled database 230 .
  • Data dialysis 700 is now described with reference to FIG. 7 .
  • reference database 220 is preferably partitioned or sorted into sets based on some criteria. These sorting criteria may vary. For example, as illustrated in table 810 of FIG. 8 , in this embodiment, data records may be sorted into sets based on last name, with the values arranged in increasing numeric order (recall that content of raw data is now represented as base-40 numbers in reference database 220 ).
  • Table 810 is derived from reference database table 620 illustrated in FIG. 6 , with each entry of table 810 defined by a unique last name and having a corresponding set of table 620 records matching that last name.
  • table 810 includes a field for defining the set (in this case, a last name), as well as identifiers for members of the set (such as indices, pointers or other appropriated references—in this case PIDNs).
  • not all vectors in reference database 220 will have data for the field on which the sets are based. Such vectors may be handled in various manners. For example, all vectors in reference database 220 having no data for that data field may be regarded as members of a single, additional set. Alternatively, each vector in reference database 220 having no data for that data field may be regarded as the single member of its own set.
  • a step 720 those data records within the partitioned sets identified as duplicates are marked.
  • duplicate data may be unnecessary and may be discarded.
  • all information remains in reference database 220 as all information, even erroneous, incomplete, or duplicate information may be better than no information and may be useful for some purpose, such as identifying fraud.
  • comparing a pair of vectors may identify duplicates.
  • Various operations may be used, as would be apparent.
  • a straightforward vector subtraction may be performed to measure the degree of similarity between two records.
  • Other techniques may be used to identify duplicate vectors such as using “look-up” tables to identify common names, nicknames, abbreviations, etc.
  • Table 810 of FIG. 8 illustrates that the last name “Smith” corresponds to PIDNs 2, 4, 8, 9, and 11, representing vectors formed from entries 2 , 4 , 8 , 9 , and 11 of the reference database tables 610 - 670 illustrated in FIG. 6 :
  • Vector (or matrix) operations comparing the vectors and thresholds for determining when two entries are similar enough to be regarded as duplicates may be defined as appropriate for various embodiments.
  • the sum of the absolute differences between corresponding coefficients of a pair of vectors may indicate a similarity between the corresponding pair of records.
  • This pair of vectors may be considered duplicates if a first vector is not inconsistent with any field of a second vector, and does not provide any additional data.
  • additional rules would also be defined, for example, for comparing entries of different lengths (e.g., right aligning character strings corresponding to numbers, and left aligning character strings corresponding to letters), for recognizing commonly misspelled or spelling variations of words, and for recognizing transposed letters in words. This processing may be performed by various mechanisms, as would be apparent.
  • none of the data records are exact duplicates, and so none are marked in step 720 .
  • the preferred embodiment of the present invention correlates data records remaining within each set and in a step 740 , further partitions the data records into independent subsets of data records.
  • the “correlation” between two vectors is a measurement of how closely one is related to the other, and specific methods of correlation may vary depending on the intended application.
  • a general discussion and examples of correlation functions may be found in references such as NUMERICAL RECIPES IN C: THE ART OF SCIENTIFIC COMPUTING (Cambridge University Press, 2nd ed. 1992) by William H. Press, et al.
  • Other techniques and examples may be found in THE ART OF COMPUTER PROGRAMMING (Addison-Wesley Pub., 1998) by Donald E. Knuth.
  • a simple measurement of the correlation between vectors is their dot product, which may be weighted as appropriate.
  • the dot product may be calculated on only a subset of the vector coefficients, or may be defined to compare not only corresponding coefficients, but also other pairs of coefficients determined to be in related fields (i.e., comparing a “first name” coefficient of a first vector with a “middle name” coefficient of a second vector).
  • the correlation function may be appropriately tailored for its intended application. For example, a correlation function may be defined to appropriately compare entries of different lengths and to appropriately distinguish between significant and insignificant differences, as would be apparent.
  • an example of a correlation function compares vectors corresponding to the members of a set sharing the same last name to identify independent subsets of vectors. Again, this determination may be based on application-specific criteria. In this example, independent vectors may be defined to be those vectors representing different individuals.
  • a correlation parameter reflecting the degree of independence of a pair of vectors is assigned. For example, a high value may be assigned to indicate a high degree of similarity, and a low value may be assigned to indicate a limited degree of similarity.
  • the correlation value is then compared to a predetermined threshold value—which again, may vary in different applications—to determine whether the two records corresponding to those vectors are considered to be independent.
  • the preferred embodiment partitions the data records into subsets of independent data records within each set.
  • members of an independent subset may be identified as those members having: the same last name (taking into consideration misspellings and spelling variations); relatively similar first names (taking into consideration misspellings, spelling variations, nicknames, and combinations of first and middle names and initials); having one or more matching account numbers; and having no more than three addresses (to allow for work and home addresses, and one change of address).
  • Results of applying such a function are illustrated in Table 820 of FIG. 8 .
  • the individuals identified are:
  • FIG. 9 illustrates a two-dimensional example of a concept referred to as clustering which is used conceptually to describe some general aspects of the present invention.
  • clusters exist as a collection of two-dimensional points. These clusters are identified as: (a,b), (c,d), (e,f), and (g,h).
  • each cluster is formed from one or more points in the two-dimensional space. Each point corresponds to a data record that represents (with more or less accuracy) the “true” value of the cluster in the space.
  • clusters (a,b,) and (c,d) are fairly easy to distinguish from one another and from clusters (e,f) and (g,h).
  • clusters (e,f) and (g,h) are not easily distinguished from one another. Extending the space (i.e., adding additional data fields to the vectors), may increase the separation between clusters such as (e,f) and (g,h) so that they become more readily distinguished from one another. Alternately, extending the space may indicate that (g,h) is a point that belongs to cluster (e,f) or even cluster (c,d). In the abstract, the space may be extended infinitely, resulting in a Hilbert space, which has various well-known characteristics. These characteristics may be exploited by the present invention for large, albeit not infinite, vectors as would be apparent.
  • deleting data fields from the vectors may also identify some correlations.
  • reducing the space may identify certain clusters that are in fact representing the same individual or other unique entity. For example, one record in a database may have ten data fields exactly identical to the same ten data fields in a second record in the database. These data fields may correspond to a first name, a birth date, an address, a mother's maiden name, etc. However, these two records may have two fields that are different. These two fields may correspond to a last name and a social security number. In some cases, these records may correspond to the same individual. The present invention simplifies the process for identifying these types of records that would be difficult, if not impossible, to detect using conventional methods.
  • removing one or more particular data fields from a vector and reducing the corresponding space may reveal clusters that otherwise would not be apparent. Doing this for data fields traditionally used for identification purposes (e.g., last name, social security number, etc.) may reveal duplicate records in databases. This may be particularly useful for identifying fraud. Removing data fields where a vector includes an empty field value for that data field may also reveal clusters that would not otherwise be apparent.
  • the best information for the individual or entity may be extracted from the information provided by each record or “black dot.”
  • the principles of the present invention may be extended beyond simple vectors and data fields.
  • the present invention may be extended through the use of tensors representing objects in a multi-dimensional space.
  • the present invention may be used to represent the parameters of various physical phenomenon to gain additional insight into their operation and effect.
  • Such application may be particularly useful for deciphering the human gene and aid in the efforts of programs such as the Human Genome Project.
  • Stranded data records are those records from reference database 220 that were not partitioned into any set in step 710 .
  • reference database 220 may include a large number of tables corresponding to data fields and a large number of vectors having data for various combinations of fields. For example, in an embodiment having a reference database 220 including 20 tables for different data fields and 1000 vectors defined by related data records for each table, suppose only 800 of those 1000 vectors have data for the field “last name,” by which the sets were created in step 710 .
  • Step 710 may not partition those 200 vectors with no “last name” data into any set, or to partition each of those 200 vectors into its own set. In either case, the result is that those 200 vectors are not correlated with any others in steps 720 , 730 , and 740 .
  • Step 750 may evaluate those vectors.
  • Methods of evaluation may vary. For example, one embodiment may correlate each stranded entry with one member of each subset identified in step 740 . Depending on the resulting correlation values, that vector may be added to the subset with which it is most highly correlated, or may define a new subset. Alternatively, in some embodiments, it may be determined that such evaluation is too time-consuming and step 750 may be completely skipped.
  • Steps 710 - 750 may be repeated as needed for specific embodiments.
  • some embodiments will have reference data 220 having a large number of fields and a large number of entries, with many entries having data for only a subset of fields. In such a case, performing steps 710 - 750 on a single field is unlikely to derive all relevant information.
  • correlating on the single field “last name” may provide only partial information about the correlation between those entries. For example, Jennifer Smith, corresponding to PIDNs 2 and 8 in FIG. 6 , may be the same individual as Jennifer Brown, corresponding to PIDN 10, because PIDNs 2 and 10 may share a common account number.
  • Performing the correlation on the last name field may not identify these PIDNs as corresponding to the same individual because they were evaluated only against other PIDNs sharing the same last name.
  • Performing a correlation on the account number field may provide additional information about whether these PIDNs are related.
  • correlation across various data fields may be necessary to fully evaluate the degree of relatedness of the data in reference database 220 .
  • reference database 220 has been distilled into a distilled database 230 , as illustrated in FIG. 2 .
  • these two databases are handled separately and coexist with one another.
  • a single database exists with records marked or otherwise identified as belonging to reference database 220 or distilled database 230 . This may be accomplished by assigning by using different ranges of PIDNs for the records in the two databases.
  • relationships between records in the two databases may be maintained by adding a constant value to the PIDN for the record in reference database 220 to generate a PIDN for the record in distilled database 230 .
  • a record with a PIDN of 12345 in reference database 220 may have a PIDN of 9012345 in distilled database 230 . In this manner, the two databases may be treated as distinct portions of a single database.
  • distilled database 230 identifies subsets of data records from the reference database 220 as related records, and as noted above, probabilities may be determined for fields in the reference database 220 to provide a qualitative measure of their completeness. This may be accomplished by assigning a probability of completeness to each of the individual data fields and then using them to compute an overall probability of completeness for the data record.
  • a value of ‘J’ may be assigned a low probability (e.g., 0 or 0.1), a value of ‘JOHN’ may be assigned a higher probability (e.g., 0.7 or 0.8), and a value of ‘JONATHAN’ may be assigned the highest probability (e.g., 0.9 or 1.0).
  • a value of ‘J’ may be assigned a low probability (e.g., 0 or 0.1)
  • a value of ‘JOHN’ may be assigned a higher probability (e.g., 0.7 or 0.8)
  • a value of ‘JONATHAN’ may be assigned the highest probability (e.g., 0.9 or 1.0).
  • distilled database 230 (which references records of the reference database 220 ) may be manipulated to provide formatted reports as needed.
  • an embodiment may be tailored to generate a report listing subsets of related records, with records of a subset providing information about a specific individual or entity.
  • the records within such a subset may provide information, for example about different fields of information; aliases and/or variations of names, addresses, social security numbers, etc., used by the individual; and fields—such as occupation, address, and account numbers—for which that individual may have more than one entry.
  • the subsets may be ordered numerically in the report.
  • the base-40 format provides the additional advantage of representing alphabetical characters as their respective letters (as illustrated in the conversion table above).
  • the report will show entries in numerical representation, that representation retains the semantic significance of the data it represents, allowing the data to be manually read and analyzed. For example, if the report shows records for an individual having entries for names including J SMITH, JOHN SMITH, JOHN G SMITH, G SMITH, and GERALD SMITH, a person reading that report would understand that this individual uses various first names, including his first name or initial, his middle name or initial, or some combination thereof.
  • new data may be added from time to time.
  • the present invention accounts for adding new (or changed) data 240 , which will affect reference database 220 and distilled database 230 .
  • new data records 240 may be formatted as described with reference to FIG. 3 , and entered into the existing reference database 220 . Additionally, new data records 240 may be measured against distilled database 230 to determine if new information or content is available in new data record 240 . For example, a new data record 240 may be correlated with data records from distilled database 230 to determine whether that new data record 240 is related to any data records already present in distilled database 230 . If so, and new data record 240 contains information or content not already present in distilled database 230 , new data record 240 may be used to update distilled database 230 . For example, if new data record 240 included information for an individual named John Smith that corresponds to data records already present in distilled database 230 but provided the additional information that Mr. Smith's middle name was Greg, that additional information may be appropriately added to distilled database 230 .
  • Changes to data records in reference database 220 and distilled database 230 may be handled using standard database protection operations, as described in references such as C. J. DATE, INTRODUCTION TO DATABASE SYSTEMS (Addison Wesley, 6th ed. 1994) (see specifically, Part IB), referenced above.
  • references such as C. J. DATE, INTRODUCTION TO DATABASE SYSTEMS (Addison Wesley, 6th ed. 1994) (see specifically, Part IB), referenced above.
  • related data records in reference database 220 are updated as determined by standard relational definitions and where appropriate, in accordance with relations defined in distilled database 230 .
  • the invention may be used to combine billing information collected from multiple sources to derive a distilled database in which related data records are recognized and duplicate and erroneous data records are eliminated. As suggested, this may be particularly useful in cases, for example, involving fraud.
  • persons using credit card or other forms of retail fraud make minor changes to certain pieces of their personal information while leaving the majority of it the same. For example, oftentimes, digits in a social security number may be transposed or an alias may be used. Often, however, other information such as the person's address, date of birth, mother's maiden name, etc., is used identically.
  • the present invention may be used as a gatekeeper for a particular database at the outset to maintain integrity of the database from the very beginning, rather than achieving integrity in the database at a later date.
  • no raw data 210 is present and only new data 240 exists.
  • new data 240 is added to the database, it is measured against distilled database 230 to determine whether new data 240 includes additional information or content. If so, only that new information or content is added to distilled database 230 by updating an existing record in distilled database 230 to reflect the new information or content as would be apparent.
  • formatting process 300 may format data using different radices or other character sets, and may use various data structures.
  • the data structures may represent multiple fields, and depending on the application, will represent a variety of fields.
  • fields may include an account status, an account number, and a legal status, in addition to personal information about the account holder.
  • fields may include various alleles or other genetic characteristics detected in tissue samples.

Abstract

A system and method for organizing raw data from one or more sources. The content of the raw data is converted into an appropriate number system and stored in a format that facilitates the use of efficient mathematical operations. The number system is selected to handle each of the various elements, characters, or other representative indicia found in the raw data. Furthermore, the number system is selected so that the numerical data retains semantic significance with respect to the raw data. Once converted into the numeric format, the data is processed using various techniques to extract the best information from the raw data into a distilled database.

Description

    CROSS-REFERENCE TO RELATED APPLICATIONS
  • This application is a continuation application of application Ser. No. 09/357,301, filed on Jul. 20, 1999, the entire content of which is hereby incorporated by reference.
  • BACKGROUND
  • 1. Field of the Invention
  • The present invention relates to database systems and more particularly, to a system and method for organizing data in a database system.
  • 2. Discussion of the Related Art
  • Computerized database systems have long been used and their basic concepts are well known. A good introduction to database systems may be found in C. J. DATE, INTRODUCTION TO DATABASE SYSTEMS (Addison Wesley, 6th ed. 1994).
  • In general, database systems are designed to organize, store and retrieve data in such a way that the data in the database is useful. For example, the data, or subsets of the data, may be searched, sorted, organized and/or combined with other data. To a large extent, the usefulness of a particular database system, is dependent on the integrity (i.e., the accuracy and/or correctness) of the data in the database system. Data integrity is affected by the degree of “disorder” in the data stored. Disorder may occur in the form of erroneous or incomplete data such as duplicate data, fragmented data, false data, etc. In many database systems, from time to time, existing data may be edited and processed, and as a result, additional errors may be introduced. In some database systems, new data may be introduced. Additionally, as database systems are upgraded with new hardware and/or software, data conversion may be required or additional fields may become necessary. Furthermore, in some applications, the data in the database may simply become outdated over time.
  • Regardless of the preventative steps taken, some degree of disorder is eventually introduced in conventional database systems. This degree of disorder increases exponentially over time until eventually, the data in a conventional database becomes entirely useless. As a result, even a small degree of disorder eventually affects the integrity of the database system.
  • Unfortunately, identifying and correcting disorder in the data are often difficult, if not impossible, tasks particularly in large database systems. Traditionally, such tasks are performed manually, making these tasks time-consuming, expensive, and subject to human error. Furthermore, due to the very nature of the task, much of the disorder may go largely undetected. What is needed is a system and method for organizing data in a database system to overcome these and other associated problems.
  • SUMMARY OF THE INVENTION
  • The present invention provides a system and method for organizing data in a database system. The present invention derives a distilled database of accurate data from raw data included in one or more raw data sources. The raw data is converted from its original format(s) to a numeric format. According to one embodiment of the present invention, the raw data is represented as a vector having numeric elements. Once the raw data is represented numerically, various mathematical operations such as correlation functions, pattern recognition methods, or other similar numeric methods, may be performed on these vectors to determine how content in a particular vector corresponds to others vectors in a “distilled” or reference database. The distilled database is formed from sets of one or more related vectors that are believed to be unique (e.g., orthogonal) with respect to the other sets. These sets represent the best information available from the raw data. After all the raw data has been incorporated into the distilled database, new data may be screened to ensure that new errors are not introduced into the distilled database. The new data may be also evaluated to determine whether it is unique or whether it includes better information than that already present in the distilled database. The new data is added to the distilled database accordingly.
  • One of the features of the present invention is that raw data is converted into a numeric format based on a number system having an appropriate radix. An appropriate radix is determined according to the type of information included in the raw data. For example, for raw data generally comprised of alpha-numeric characters, an appropriate radix may be greater than or equal to the number of different alpha-numeric characters present in the raw data. Using such a number system allows raw data to be represented numerically, allowing for manipulation through various well-known mathematical operations.
  • Another feature of the present invention is that the number system may be selected so that the numbers themselves retain semantic significance to the raw data they represent. In other words, the numerals in the number system are selected so that they correspond to the raw data. For example, in the case of raw data comprised of alphanumeric characters, the numerals are selected to correspond to the alphanumeric characters they represent. When the numerals in the number system are subsequently displayed, they appear as the alphanumeric characters they represent.
  • Another feature of the present invention is that once the raw data is represented as vectors in an appropriate number system, the represented data may be efficiently manipulated in the database (e.g., sorted, etc.) using various well-known techniques. Furthermore, various well-known mathematical operations may be performed on the vectors to analyze the data content. These mathematical operations may include correlation functions, eigenvector analyses, pattern recognition methods, and others as would be apparent.
  • Still another feature of the present invention is that the raw data is incorporated into a distilled database. The distilled database represents the best information extracted from the raw data without having any data disorder.
  • Yet another feature of the present invention is that new data may be compared to the distilled database to determine whether the new data actually includes any new information or content not already present in the distilled database. Any new information not already in the distilled database is added to the distilled database without adding any disorder. In this manner, the integrity of the distilled database may be maintained.
  • Other features and advantages of the invention will become apparent from the following drawings and description.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The present invention is described with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears.
  • FIG. 1 illustrates a processing system in which the present invention may be implemented.
  • FIG. 2 illustrates stages of data processed by one embodiment of the present invention.
  • FIG. 3 is a flow diagram for converting raw data from its original format into a numeric format in accordance with one embodiment of the present invention.
  • FIG. 4 illustrates a data record suitable for use with the present invention.
  • FIG. 5 illustrates raw data tables suitable for use with the present invention.
  • FIG. 6 illustrates reference data tables, representing data formatted in accordance with an embodiment of the present invention.
  • FIG. 7 is a flow diagram for analyzing reference data in accordance with an embodiment of the present invention.
  • FIG. 8 illustrates distilled data table, representing related data correlated in accordance with an embodiment of the present invention.
  • FIG. 9 illustrates an example of data clustering in a two-dimensional space.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • The present invention is directed to a system and method for organizing data in a database system. The present invention is described below with respect to various exemplary embodiments, particularly with respect to various database applications. However, various features of the present invention may be extended to other areas as would be apparent. In general, the present invention may be applicable to many database applications where large amounts of seemingly unrelated data must be compiled, stored, manipulated, and/or analyzed to determine the various relationships present in the content represented by the data. More particularly, the present invention provides a method for achieving and maintaining the integrity (i.e., accuracy and correctness) of data in a database system, even when that data initially possesses a high degree of disorder. As used herein, disorder refers to data that is duplicative, erroneous, incomplete, imprecise, false or otherwise incorrect or redundant. Disorder may present itself in the database system in many ways as would be apparent.
  • One embodiment of the present invention is used to maintain a database associated with accounts receivable. In this embodiment, a company may collect data relating to various persons, businesses and/or accounts from one or more sources. These sources may include, for example, credit card companies, financial institutions, banks, retail, and wholesale businesses and other such sources. While each of these sources may provide data relating to various accounts, each source may provide data representing different information based on its own needs. Furthermore, this data may be organized in entirely different ways. For example, a wholesale distributor may have data corresponding to accounts receivable corresponding to business accounts. Such data may be organized by account numbers, with each data record having data fields identifying an account number, a business associated with that account number, an address of that business, and an amount owed on the account. A retail company may have data records representing similar information but based on accounts corresponding to individuals as well as businesses.
  • In other embodiments of the present invention, other types of sources may provide different types of data. For example, the scientific institutions may provide scientific data with respect to various areas of research. Industrial companies may provide industrial data with respect to raw materials, manufacturing, production, and/or supply. Courts or other types of legal institutions may provide legal data with respect to legal status, judgments, bankruptcy, and/or liens. As would be apparent, the present invention may use data from a wide variety of sources.
  • In another embodiment of the present invention, a database may be maintained to implement an integrated billing and order control system. In addition to billing-type information from sources similar to those described above, this embodiment may include data records corresponding to inventory, data records corresponding to suppliers of the inventory, and data records corresponding to purchasers of the inventory. Inventory data may be organized by part numbers, with each data record having data fields identifying an internal part number, an external part number (i.e., supplier part number), a quantity on hand, a quantity expected to ship, a quantity expected to be received, a wholesale price, and a retail price. Supplier data may be organized by a supplier number; and customer data may be organized by a customer number. Data records corresponding to each of these records may include data fields identifying a part number, a part price, a quantity ordered, a ship data, and other such information.
  • Another embodiment of the present invention may include an enterprise storage system that consolidates corporate information from multiple, dissimilar sources and makes that information available to users on the corporate network regardless of the type of the data, the type of computer that generated the data, or the type of computer that requested the data. Still another embodiment of the present invention includes a business intelligence system that warehouses and markets information and allows that information to be processed and analyzed on-line.
  • The present invention enables raw data collected from different sources to be analyzed and distilled into a collection of accurate data, organized in a way that is useful for a particular application. Using the above example of an integrated billing and order control system, explained more fully below, the present invention may produce a distilled database in which related data, such as data relating to a particular supplier or customer, may be identified as such. In this example, duplicate data corresponding to the same supplier or customer may be identified and/or discarded, and erroneous data associated with the supplier or customer may be identified, analyzed, and possibly corrected.
  • In general, the present invention may be implemented in hardware or software, or a combination of both. Preferably, the present invention is implemented as a software program executing in a programmable processing system including a processor, a data storage system, and input and output devices. An example of such a system 100 is illustrated in FIG. 1. System 100 may include a processor 110, a memory 120, a storage device 130, and an I/O controller 140, coupled to one another by a processor bus 150. I/O controller 140 is also coupled via an 110 bus 160 to various input and output devices, such as a keyboard 170, a mouse 180, and a display 190. Other components may be included in the system 100 as would be apparent.
  • FIG. 2 illustrates various forms of data processed by the present invention. Raw data 210 may be collected from one or more sources, such as raw data 210A and raw data 210B. As used herein, “raw data” simply refers to data as it is received from a particular source. Additional sources of raw data 210 may be included as would be apparent. As explained below, raw data 210 from various sources is converted into a numerical format and stored in a reference database 220. Using a process referred to herein as “data dialysis,” the present invention “purifies” raw data 210 to form reference data in reference database 220. Reference database 220 includes all the information found in raw data 210 including duplicate, incomplete, inconsistent, and erroneous data.
  • Distilled data stored in a distilled database 230 is derived from the reference data of reference database 220. Distilled data represents the “accurate” data available from raw data 210. Distilled database 230 includes the unique information found in raw data 210. Distilled data thus represents the best information available from raw data 210.
  • As also explained below, the present invention further provides for using distilled database 230 to analyze and verify new data 240, which may also be used to update the reference database 220 and distilled database 230 as appropriate.
  • While the present invention has numerous embodiments, to clarify its description, a preferred embodiment is explained with reference to FIGS. 3-8 in a context of an integrated billing and order control system. In this embodiment, raw data 210 is a collection of data collected from various sources, such as order processing, shipping, receiving, accounts payable and accounts receivable, etc. This raw data 210 may include data records that are related but have different data fields, duplicate data records, data records having one or more erroneous data fields, etc. To address such errors, the present invention converts raw data 210 from their original formats and data structures (which may vary based on the source) into a numeric format and stores this reference data in reference database 220.
  • According to the present invention, the reference data is then compared and analyzed to distill the best information available. In one embodiment of the present invention, this best information may be stored as distilled data in distilled database 230. This process is now described.
  • Collecting Raw Data
  • FIG. 3 illustrates the process by which raw data 210 is converted into reference data in reference database 220 according to one embodiment of the present invention. In a step 310, raw data 210 is collected from a raw data source. As illustrated in FIG. 2, raw data 210 may include data from one or more sources such as raw data 210A and raw 210B. As used herein, “data” refers to the physical digital representation of information, and data “content” refers to the meaning of, or information included in or represented by that data. The different records in raw data 210 may include similar types of data content. For example, in a billing context, different records in raw data 210 may all include data content relating to a particular account.
  • Raw data 210 will typically be received in the form of data records 400, as illustrated in FIG. 4. Each data record 400 generally includes related information, such as information for a specific individual, company, or account. Each data record 400 stores this information in one or more data fields 410. Examples of possible data fields 410 include, for example, an account number, a last name, a first name, a company name, an account balance, etc. Each data field 410, in turn, may include one or more data elements 420 for representing information for that specific record and specific field. Data elements 420 may exist in various formats, such as alphanumeric, numeric, ASCII, and EBCDIC, or other representation as would be apparent. Raw data 210 collected from different sources may be formatted differently. Data records 400 may include different data fields 410, and the information included in data fields 410 may be represented using data elements 420 in different formats, as would also be apparent.
  • Examples of raw data 210 are illustrated in raw data tables 510, 520, and 530 of FIG. 5. Data records, such as data record 510-1 and data record 510-2, are illustrated as rows of raw data tables 510, 520, and 530, whereas data fields, such as data field 510-A and data field 510-B, are illustrated as columns of raw data tables 510, 520, and 530. The tables illustrated in FIG. 5 are examples of data that might be found in various embodiments of the present invention. In other embodiments, data may come from many sources and may be formatted as databases having a much larger number of data records and/or data fields, as would be apparent.
  • Conversion to Numeric Format
  • Referring to FIG. 3, in a step 320, the present invention converts raw data 210 from its original representation (which may be in alphanumeric, numeric, ASCII, EBCDIC, or other similar formats) to a numeric representation. This ensures that reference data is represented in the same manner. Thus, the reference data, including that data from different sources, may be similarly processed.
  • According to the present invention, raw data 210 is converted from its original representation into an appropriate numeric representation. An appropriate numeric representation uses a number system in which each possible value of data element 420 may be represented by a unique digit or value in the number system. In other words, a radix for the number system is selected such that the radix is at least as great as the number of possible values for a particular data element. For example, in a biotechnology application for detecting nucleotide sequences of Adenine (A), Guanine (G), Cytosine (C), and Thymine (T) in nucleic acids, each data element may be one of only four values: A, G, C, and T. In such an application, a radix of four for the number system may be sufficient to represent each data element as a unique number. One such number system may include the numbers A, G, C, and T. In some embodiments of the present invention, it may be desirable to use a radix at least one greater than the number of different possible value of data element 420 in order to provide a number representative of an empty field. In this case, such as number system may include the numbers A, G, C, T, and ̂, where ̂ is the empty field value.
  • According to a preferred embodiment of the present invention, data elements 420 in raw data 210 are comprised of characters such as alphanumeric characters. In this preferred embodiment, a radix of 40 is selected to represent the alphanumeric characters as illustrated in the table below. (Note that a minimum radix of 36 is required.) This radix is selected to accommodate the ten numeric characters “0”-“9” and the twenty-six alphabetic characters “A” to “Z” as well as to allow for several additional characters. In this embodiment, uppercase and lowercase characters are not distinguished from one another.
  • As illustrated in Table 1, the base-40 number system includes the numbers 0-9, followed by A-Z, further followed by four additional numbers. One of these numbers may used to represent an empty field. This number is used to represent a data field 410 that is empty or has no value (in contrast to a zero value). Other numbers may be used, for example, to represent other types of information such as spaces or used as control information.
  • TABLE 1
    Alpha- Base-10 Base-40 Alpha- Base 10 Base-40
    Numeric Number Number Numeric Number Number
    0 0 0 K or k 20 K
    1 1 1 L or 1 21 L
    2 2 2 M or m 22 M
    3 3 3 N or n 23 N
    4 4 4 O or o 24 O
    5 5 5 P or p 25 P
    6 6 6 Q or q 26 Q
    7 7 7 R or r 27 R
    8 8 8 S or s 28 S
    9 9 9 T or t 29 T
    A or a 10 A U or u 30 U
    B or b 11 B V or v 31 V
    C or c 12 C W or w 32 W
    D or d 13 D X or x 33 X
    E or e 14 E Y or y 34 Y
    F or f 15 F Z or z 35 Z
    G or g 16 G 36 [
    H or h 17 H 37 \
    I or i 18 I 38 ]
    J or j 19 J 39 {circumflex over ( )}
  • Representation of raw data 210 in a base-40 format has numerous benefits. One benefit is that raw data 210 may be represented in a numeric fashion, facilitating straightforward mathematical manipulation. Another benefit is that proper selection of both the radix and the numerals in the number system allows the represented content to maintain semantic significance, facilitating recognition the content of raw data 210 in its representation in the numeric format. For example, the word “JOHN” represented by the four alphanumeric characters “J” “O” “H” “N” may be represented in various number systems. One such number system is a base-40 number system. Using Table 1, representing the alphanumeric characters “JOHN” as a base-40 number would result in the “tetradecimal” value ‘JOHN’, which is equivalent to the decimal value 1,255,103 (19*403+24*402+17*401+23*400, where base-40 ‘J’ equals decimal 19, etc.). Note that the base-10 number loses semantic significance from the content of raw data 210 whereas the base-40 number retains semantic significance, as the number ‘JOHN’ is recognizable as the content “JOHN.” Semantic significance provides the benefits of a numeric representation while maintaining the ability to convey semantic content.
  • In some embodiments of the present invention, the selection of a radix and its corresponding number system may depend upon the number of bits used by processor 110. The number of bits used by processor 110 and the radix chosen for the number system define the number characters that can be represented by a data word in processor 110. This relationship is governed according to the following equation:

  • N=B*ln(2)/ln(R),
  • where N is the number of whole characters (i.e., fractional characters are discarded) represented by a data word of processor 110, B is the number of bits per data word, and R is the selected radix. This relationship limits the number of data elements 420 of raw data 210 that may fit in a data word. For example, in a 32-bit machine, the maximum number of characters that may fit in a data word using a base-40 number system is six (32*ln(2)/ln(40)=6.013). The maximum number of characters that may fit in a data word using a base-41 number system is only five (32*ln(2)/ln(41)=5.973). Thus, in some embodiments of the present invention, in addition to having a radix sufficiently large to maintain semantic significance, the radix may also be selected to maximize the number of characters represented by a single data word. In the embodiment with raw data comprised of alphanumeric characters, an appropriate radix may range from 36 to 40. This range maintains semantic significance while maximizing the number of characters represented by the 32-bit data word. Other types of raw data and other sizes of data word may dictate other appropriate radix ranges in other embodiments of the present invention.
  • The embodiment of the present invention described above does not distinguish between uppercase and lowercase characters. However, other embodiments of the present invention may distinguish between these types of characters. Accordingly, a base-64 representation (“0”-“9”, “A-”Z″, “a”-“z”, and two other values) may be appropriate to distinguish between these characters as would be apparent.
  • The number of data elements 420 in each data field 410 also dictates the precision required by the number as represented in processor 110. As described above, each data field 410 may only be six characters or data elements 420 wide for single precision operations in a 32-bit machine. In some embodiments of the present invention, this may be insufficient. In these embodiments, double, triple, or even quadruple precision may be required to represent the entire data field 410 as a single value. Double precision numbers are sufficient for up to twelve character data fields 410; triple precision numbers are sufficient for up to eighteen characters; and quadruple precision numbers are sufficient for up to twenty-four characters.
  • Alternate embodiments of the present invention may accommodate large data fields by breaking a large data field into one or more smaller data fields. The large data fields may be broken at boundaries defined by spaces. For example, a data field representing an address such as “123 West Main Street” may be broken into four smaller data fields: ‘123’, ‘West’, ‘Main’, and ‘Street’. The large data fields may also be broken at data word boundaries. In the address example above, the smaller data fields might be: ‘123We’, ‘st\Mai’, ‘n\Stre’, and ‘et’, where the number ‘\’ is used to represent a space. Other embodiments of the present invention may accommodate large data fields in other manners as would be apparent.
  • Data Structure Conversion
  • As illustrated in FIG. 3, in a step 330, raw data 210 represented as a number is stored in a predefined data structure. In one embodiment of the present invention, this data structure is a single-field table as illustrated by Tables 610-670 of FIG. 6. This data structure may vary. For example, in other embodiments of the present invention, the data structure may be a multiple-field table instead of a single-field table. In these embodiments, the data structures may be implemented with standard features such as table headers and indices, and as explained in greater detail below, may also include probability values for each record. These probability values represent the likelihood that the data in that record is complete. Higher probability values may indicate a higher probability of completeness, and lower probability values similarly may indicate a lower probability of completeness. This is described in further detail below. Initially, the probability values are set to 0. Other embodiments may also include key numbers or identification numbers to aid in sorting and in maintaining relationships among the data records.
  • In a preferred embodiment of the present invention, raw data 210 illustrated in FIG. 5 includes three tables 510, 520, and 530. Table 510 may represent raw data 210 from, for example, a company's accounts receivable system. Columns of table 510 represent data fields for an account number, a last name, a first initial, and additional fields for listing various orders processed for a particular individual. Rows of table 510 (such as 510-1 and 510-2) represent data records for different individuals. Tables 520 and 530 may represent raw data 210 maintained by credit card companies. Columns of tables 520 and 530 represent data fields for an account number, a last name, a first name, and an address. Rows of tables 520 and 530 represent data records for specific accounts.
  • In the preferred embodiment, step 330 converts raw data 210 from the format illustrated in FIG. 5 into a format illustrated in FIG. 6. FIG. 6 illustrates raw data 210, combined from the various raw data tables 510, 520, 530 of FIG. 5, represented as numbers in a base-40 number system, and formatted as new tables (tables 610-670), which together may comprise reference database 220.
  • Each reference database table 610-670 corresponds to an individual field from raw data tables 510, 520, and 530 of FIG. 5. More specifically, data records of reference data tables 610-670 correspond to the data records of raw data table 510, followed by the data records of raw data table 520, followed by the data records of raw data table 530. In one embodiment of the present invention, where a raw data table record has no information for a particular data field 410 represented in a reference table 610-670, a empty field value is entered in that field in the reference table. For example, the first data record 510-1 of Table 510 has no information about an address, and thus an empty field value is placed in the first position of table 670.
  • Data is preferably stored in reference database 220 in such a way that all data corresponding to a single data record in a raw data table is readily identified. In the embodiment represented in FIGS. 5 and 6, for example, data corresponding to any specific data record of the raw data tables (tables 510, 520, 530) is preferably represented in reference tables 610-670 as a “vector” of numeric data stored at an index i across reference tables 610-670. For example, data corresponding to the sixth record 520-6 of raw data table 520 (illustrated as account number “A60” belonging to “Jennifer Brown,” residing at “51 Fourth Street”) is represented in reference database tables 610-670 as a vector having coefficients formed from the tenth records 610-10, 620-10, 630-10, 640-10, 650-10, 660-10, and 670-10 of the tables 610-670.
  • As illustrated in FIG. 6, reference database 220 includes a new table 610 that does not correspond to any data field 410 in raw data 210 illustrated in FIG. 5. This table is a “key table” that identifies the related data in these data vectors. As described below, reference database 220 comprised of the tables illustrated in FIG. 6 may include additional key tables for data fields. These may include a personal identification number (“P IDN”), an account identification number (“AIDN”), or other types of identification numbers. These key tables or identification numbers may be used to identify sets of related data vectors in reference database 220.
  • In this example, key table 610 has a single field “PIDN,” which stands for personal identification number. Key table 610 provides a unique identifier such that a specific PIDN number never refers to more than one person represented in raw data 210. In other words, the PIDN number reflects the fact that many multiple records in raw data 210 may refer to the same person.
  • Preferably, each data record in the key table 610 initially corresponds to a different data record represented in the raw data tables 510, 520, and 530. For example, in FIG. 6, data record 610-10 in the key table 610 is implemented such that it includes identifiers (such as pointers or indices) for corresponding data in reference tables 620-670, which together corresponds to a single record 520-6 in raw data table 520.
  • Initially, while a single PIDN does not refer to multiple individuals, a single individual may correspond to multiple PIDNs. For example, in FIG. 6, vector 4 (defined by PIDN 4) and vector 9 (defined by PIDN 9) appear to refer to the same person, but as illustrated, this person is initially assigned to two PIDN numbers—PIDN 4 and PIDN 9. As described below, the present invention enables a determination whether PIDN 4 and PIDN 9 do, in fact, refer to the same individual, and if so, assigns a single PIDN to this individual. Alternatively, some embodiments may assign a new PIDN number to individuals so determined and a reference to the old PIDN number may be retained.
  • As discussed above, in this embodiment, records are represented in the reference database tables 610-670 as vectors having coefficients of base-40 numbers across eight one-field tables. This numeric representation allows the data to be analyzed using straightforward mathematical operations that may be used to, for example, produce correlations, calculate eigenvectors, perform various coordinate transformations, and utilize various pattern recognition analyses. These operations may, in turn, be used to provide or derive information about the records and their relationships to one another. By using small, one-field tables, these operations may be performed quickly. In addition, as will be illustrated, representation in base-40 numbers with raw data 210 including alphanumeric characters allows content of raw data 210 to retain its semantic significance.
  • Data Dialysis
  • Referring back to FIG. 2, once reference database 220 is created as illustrated in FIG. 6, a data dialysis process 700 is applied to distill the most accurate data for inclusion in distilled database 230. Data dialysis 700 is now described with reference to FIG. 7.
  • Partitioning the Reference Data
  • In a step 710, reference database 220 is preferably partitioned or sorted into sets based on some criteria. These sorting criteria may vary. For example, as illustrated in table 810 of FIG. 8, in this embodiment, data records may be sorted into sets based on last name, with the values arranged in increasing numeric order (recall that content of raw data is now represented as base-40 numbers in reference database 220). Table 810 is derived from reference database table 620 illustrated in FIG. 6, with each entry of table 810 defined by a unique last name and having a corresponding set of table 620 records matching that last name. In the representation illustrated, table 810 includes a field for defining the set (in this case, a last name), as well as identifiers for members of the set (such as indices, pointers or other appropriated references—in this case PIDNs).
  • In some embodiments of the present invention, not all vectors in reference database 220 will have data for the field on which the sets are based. Such vectors may be handled in various manners. For example, all vectors in reference database 220 having no data for that data field may be regarded as members of a single, additional set. Alternatively, each vector in reference database 220 having no data for that data field may be regarded as the single member of its own set.
  • Identifying Duplicate Data
  • Returning to FIG. 7, in a step 720, those data records within the partitioned sets identified as duplicates are marked. In some embodiments of the present invention, duplicate data may be unnecessary and may be discarded. In other embodiments, all information remains in reference database 220 as all information, even erroneous, incomplete, or duplicate information may be better than no information and may be useful for some purpose, such as identifying fraud.
  • In some embodiments of the present invention, comparing a pair of vectors may identify duplicates. Various operations may be used, as would be apparent. In a simple example, a straightforward vector subtraction may be performed to measure the degree of similarity between two records. Other techniques may be used to identify duplicate vectors such as using “look-up” tables to identify common names, nicknames, abbreviations, etc.
  • Table 810 of FIG. 8 illustrates that the last name “Smith” corresponds to PIDNs 2, 4, 8, 9, and 11, representing vectors formed from entries 2, 4, 8, 9, and 11 of the reference database tables 610-670 illustrated in FIG. 6:
      • For PIDN 2: [SMITH, J, 98-002, A40, A60, ̂]
      • For PIDN 4: [SMITH, J, 98-004, A50, B10, ̂]
      • For PIDN 8: [SMITH, Jennifer, ̂, A40, ̂, 300 Pine St.]
      • For PIDN 9: [SMITH, John, ̂, A50, ̂, 37 Hunt Dr.]
      • For PIDN 11: [SMITH, Jhon, ̂, B10, ̂, 85 Belmont Ave.]
  • Vector (or matrix) operations comparing the vectors and thresholds for determining when two entries are similar enough to be regarded as duplicates may be defined as appropriate for various embodiments. In a simple example, the sum of the absolute differences between corresponding coefficients of a pair of vectors may indicate a similarity between the corresponding pair of records. This pair of vectors may be considered duplicates if a first vector is not inconsistent with any field of a second vector, and does not provide any additional data. In this embodiment, additional rules would also be defined, for example, for comparing entries of different lengths (e.g., right aligning character strings corresponding to numbers, and left aligning character strings corresponding to letters), for recognizing commonly misspelled or spelling variations of words, and for recognizing transposed letters in words. This processing may be performed by various mechanisms, as would be apparent. In the example of Table 810 of FIG. 8, none of the data records are exact duplicates, and so none are marked in step 720.
  • Correlating Data
  • Referring back to FIG. 7, in a step 730, the preferred embodiment of the present invention correlates data records remaining within each set and in a step 740, further partitions the data records into independent subsets of data records. In general, the “correlation” between two vectors is a measurement of how closely one is related to the other, and specific methods of correlation may vary depending on the intended application. A general discussion and examples of correlation functions may be found in references such as NUMERICAL RECIPES IN C: THE ART OF SCIENTIFIC COMPUTING (Cambridge University Press, 2nd ed. 1992) by William H. Press, et al. Other techniques and examples may be found in THE ART OF COMPUTER PROGRAMMING (Addison-Wesley Pub., 1998) by Donald E. Knuth.
  • As an example, a simple measurement of the correlation between vectors is their dot product, which may be weighted as appropriate. Depending on the application, the dot product may be calculated on only a subset of the vector coefficients, or may be defined to compare not only corresponding coefficients, but also other pairs of coefficients determined to be in related fields (i.e., comparing a “first name” coefficient of a first vector with a “middle name” coefficient of a second vector). As with the operations for identifying duplicate data, the correlation function may be appropriately tailored for its intended application. For example, a correlation function may be defined to appropriately compare entries of different lengths and to appropriately distinguish between significant and insignificant differences, as would be apparent.
  • In the embodiment explained with reference to the tables of FIGS. 5, 6, and 8, an example of a correlation function compares vectors corresponding to the members of a set sharing the same last name to identify independent subsets of vectors. Again, this determination may be based on application-specific criteria. In this example, independent vectors may be defined to be those vectors representing different individuals.
  • As a result of applying the correlation function, a correlation parameter reflecting the degree of independence of a pair of vectors is assigned. For example, a high value may be assigned to indicate a high degree of similarity, and a low value may be assigned to indicate a limited degree of similarity. The correlation value is then compared to a predetermined threshold value—which again, may vary in different applications—to determine whether the two records corresponding to those vectors are considered to be independent.
  • Based on the correlation values, in a step 740, the preferred embodiment partitions the data records into subsets of independent data records within each set. In the examples of FIGS. 5, 6, and Table 810 of FIG. 8, members of an independent subset may be identified as those members having: the same last name (taking into consideration misspellings and spelling variations); relatively similar first names (taking into consideration misspellings, spelling variations, nicknames, and combinations of first and middle names and initials); having one or more matching account numbers; and having no more than three addresses (to allow for work and home addresses, and one change of address).
  • Results of applying such a function are illustrated in Table 820 of FIG. 8. The individuals identified are:
      • Jennifer Brown, PIDN 10;
      • Howard Lee, PIDNs 3 and 6;
      • Carole Lee, PIDN 7;
      • Jennifer Smith, PIDNs 2 and 8;
      • John Smith, PIDNs 4 and 11;
      • John Smith, PIDN 9;
      • Ann Zane, PIDNs 1, 5, and 12; and
      • Molly Zane, PIDN 13.
  • Other operations for correlating the vectors are available. These may include computing dot products, cross products, lengths, direction vectors, and a plethora of other functions and algorithms used for evaluation according to well-known techniques.
  • FIG. 9 illustrates a two-dimensional example of a concept referred to as clustering which is used conceptually to describe some general aspects of the present invention. In FIG. 9, four clusters exist as a collection of two-dimensional points. These clusters are identified as: (a,b), (c,d), (e,f), and (g,h). As illustrated, each cluster is formed from one or more points in the two-dimensional space. Each point corresponds to a data record that represents (with more or less accuracy) the “true” value of the cluster in the space. As illustrated, clusters (a,b,) and (c,d) are fairly easy to distinguish from one another and from clusters (e,f) and (g,h). However, in this simple example, clusters (e,f) and (g,h) are not easily distinguished from one another. Extending the space (i.e., adding additional data fields to the vectors), may increase the separation between clusters such as (e,f) and (g,h) so that they become more readily distinguished from one another. Alternately, extending the space may indicate that (g,h) is a point that belongs to cluster (e,f) or even cluster (c,d). In the abstract, the space may be extended infinitely, resulting in a Hilbert space, which has various well-known characteristics. These characteristics may be exploited by the present invention for large, albeit not infinite, vectors as would be apparent.
  • Furthermore, while adding additional data fields to the vectors (i.e., extending the space) may separate clusters from one another to aid in their correlation, deleting data fields from the vectors (i.e., reducing the space) may also identify some correlations. In some embodiments of the present invention, reducing the space may identify certain clusters that are in fact representing the same individual or other unique entity. For example, one record in a database may have ten data fields exactly identical to the same ten data fields in a second record in the database. These data fields may correspond to a first name, a birth date, an address, a mother's maiden name, etc. However, these two records may have two fields that are different. These two fields may correspond to a last name and a social security number. In some cases, these records may correspond to the same individual. The present invention simplifies the process for identifying these types of records that would be difficult, if not impossible, to detect using conventional methods.
  • Thus, removing one or more particular data fields from a vector and reducing the corresponding space may reveal clusters that otherwise would not be apparent. Doing this for data fields traditionally used for identification purposes (e.g., last name, social security number, etc.) may reveal duplicate records in databases. This may be particularly useful for identifying fraud. Removing data fields where a vector includes an empty field value for that data field may also reveal clusters that would not otherwise be apparent.
  • Furthermore, once the clusters are identified as representing the same individual or entity, the best information for the individual or entity may be extracted from the information provided by each record or “black dot.”
  • The principles of the present invention may be extended beyond simple vectors and data fields. For example, the present invention may be extended through the use of tensors representing objects in a multi-dimensional space. In this manner, the present invention may be used to represent the parameters of various physical phenomenon to gain additional insight into their operation and effect. Such application may be particularly useful for deciphering the human gene and aid in the efforts of programs such as the Human Genome Project.
  • Handling Stranded Data
  • Referring again to FIG. 7, in a step 750, the preferred embodiment of the present invention evaluates “stranded” data records. Stranded data records are those records from reference database 220 that were not partitioned into any set in step 710. In some embodiments, reference database 220 may include a large number of tables corresponding to data fields and a large number of vectors having data for various combinations of fields. For example, in an embodiment having a reference database 220 including 20 tables for different data fields and 1000 vectors defined by related data records for each table, suppose only 800 of those 1000 vectors have data for the field “last name,” by which the sets were created in step 710. Step 710 may not partition those 200 vectors with no “last name” data into any set, or to partition each of those 200 vectors into its own set. In either case, the result is that those 200 vectors are not correlated with any others in steps 720, 730, and 740. Step 750 may evaluate those vectors.
  • Methods of evaluation may vary. For example, one embodiment may correlate each stranded entry with one member of each subset identified in step 740. Depending on the resulting correlation values, that vector may be added to the subset with which it is most highly correlated, or may define a new subset. Alternatively, in some embodiments, it may be determined that such evaluation is too time-consuming and step 750 may be completely skipped.
  • Repeating the Correlation Process
  • Steps 710-750 may be repeated as needed for specific embodiments. As noted above, some embodiments will have reference data 220 having a large number of fields and a large number of entries, with many entries having data for only a subset of fields. In such a case, performing steps 710-750 on a single field is unlikely to derive all relevant information. Even in the simple example explained with reference to FIGS. 5, 6, and 8, correlating on the single field “last name” may provide only partial information about the correlation between those entries. For example, Jennifer Smith, corresponding to PIDNs 2 and 8 in FIG. 6, may be the same individual as Jennifer Brown, corresponding to PIDN 10, because PIDNs 2 and 10 may share a common account number. Performing the correlation on the last name field may not identify these PIDNs as corresponding to the same individual because they were evaluated only against other PIDNs sharing the same last name. Performing a correlation on the account number field may provide additional information about whether these PIDNs are related.
  • Thus, correlation across various data fields may be necessary to fully evaluate the degree of relatedness of the data in reference database 220.
  • Using Correlation Results to Update Reference Data
  • Once steps 710-760 are completed, reference database 220 has been distilled into a distilled database 230, as illustrated in FIG. 2. In some embodiments of the present invention, these two databases are handled separately and coexist with one another. In other embodiments of the present invention, a single database exists with records marked or otherwise identified as belonging to reference database 220 or distilled database 230. This may be accomplished by assigning by using different ranges of PIDNs for the records in the two databases. Furthermore, relationships between records in the two databases may be maintained by adding a constant value to the PIDN for the record in reference database 220 to generate a PIDN for the record in distilled database 230. For example, a record with a PIDN of 12345 in reference database 220 may have a PIDN of 9012345 in distilled database 230. In this manner, the two databases may be treated as distinct portions of a single database.
  • Using the Distilled Data
  • Once data dialysis process 700 is complete, distilled database 230 identifies subsets of data records from the reference database 220 as related records, and as noted above, probabilities may be determined for fields in the reference database 220 to provide a qualitative measure of their completeness. This may be accomplished by assigning a probability of completeness to each of the individual data fields and then using them to compute an overall probability of completeness for the data record. For example, for a data field representing a first name, a value of ‘J’ may be assigned a low probability (e.g., 0 or 0.1), a value of ‘JOHN’ may be assigned a higher probability (e.g., 0.7 or 0.8), and a value of ‘JONATHAN’ may be assigned the highest probability (e.g., 0.9 or 1.0). These values may be assigned somewhat arbitrarily. However, these values help identify which data fields in the set are most likely to include the most complete information or in other words, the most probable data.
  • Use of the present invention may determine a significant amount of information about the records and their relationship to each other, and may be specifically tailored for particular applications. Furthermore, using standard database operations, distilled database 230 (which references records of the reference database 220) may be manipulated to provide formatted reports as needed. For example, an embodiment may be tailored to generate a report listing subsets of related records, with records of a subset providing information about a specific individual or entity. The records within such a subset may provide information, for example about different fields of information; aliases and/or variations of names, addresses, social security numbers, etc., used by the individual; and fields—such as occupation, address, and account numbers—for which that individual may have more than one entry.
  • Recalling that all data is represented in numerical base-40 format, the subsets may be ordered numerically in the report. The base-40 format provides the additional advantage of representing alphabetical characters as their respective letters (as illustrated in the conversion table above). Thus, while the report will show entries in numerical representation, that representation retains the semantic significance of the data it represents, allowing the data to be manually read and analyzed. For example, if the report shows records for an individual having entries for names including J SMITH, JOHN SMITH, JOHN G SMITH, G SMITH, and GERALD SMITH, a person reading that report would understand that this individual uses various first names, including his first name or initial, his middle name or initial, or some combination thereof.
  • Adding New Data
  • As with conventional database applications, new data may be added from time to time. As illustrated in FIG. 2, the present invention accounts for adding new (or changed) data 240, which will affect reference database 220 and distilled database 230.
  • Generally, new data records 240 may be formatted as described with reference to FIG. 3, and entered into the existing reference database 220. Additionally, new data records 240 may be measured against distilled database 230 to determine if new information or content is available in new data record 240. For example, a new data record 240 may be correlated with data records from distilled database 230 to determine whether that new data record 240 is related to any data records already present in distilled database 230. If so, and new data record 240 contains information or content not already present in distilled database 230, new data record 240 may be used to update distilled database 230. For example, if new data record 240 included information for an individual named John Smith that corresponds to data records already present in distilled database 230 but provided the additional information that Mr. Smith's middle name was Greg, that additional information may be appropriately added to distilled database 230.
  • Changes to data records in reference database 220 and distilled database 230 may be handled using standard database protection operations, as described in references such as C. J. DATE, INTRODUCTION TO DATABASE SYSTEMS (Addison Wesley, 6th ed. 1994) (see specifically, Part IB), referenced above. For example, in the case that changes are made to reference database 220 by an authorized database administrator, related data records in reference database 220 are updated as determined by standard relational definitions and where appropriate, in accordance with relations defined in distilled database 230.
  • Various embodiments of the present invention may be used for many different applications, some of which have been described and/or alluded to above. For example, in the application described above, the invention may be used to combine billing information collected from multiple sources to derive a distilled database in which related data records are recognized and duplicate and erroneous data records are eliminated. As suggested, this may be particularly useful in cases, for example, involving fraud. Typically, persons using credit card or other forms of retail fraud make minor changes to certain pieces of their personal information while leaving the majority of it the same. For example, oftentimes, digits in a social security number may be transposed or an alias may be used. Often, however, other information such as the person's address, date of birth, mother's maiden name, etc., is used identically. These types of fraud are readily identified by the present invention, even though they are difficult to identify by human analyses.
  • Other possible applications include uses in telemarketing, to compile a list of targeted individuals or addresses, or in mail-order catalogs, to reduce a number of catalogs sent to the same individual or family. Still another potential application is in the medical research or diagnostics fields, in which nucleotide sequences of Adenine (A), Guanine (G), Cytosine (C), and Thymine (T) in nucleic acids may be identified.
  • In other embodiments, the present invention may be used as a gatekeeper for a particular database at the outset to maintain integrity of the database from the very beginning, rather than achieving integrity in the database at a later date. In these embodiments, no raw data 210 is present and only new data 240 exists. Before new data 240 is added to the database, it is measured against distilled database 230 to determine whether new data 240 includes additional information or content. If so, only that new information or content is added to distilled database 230 by updating an existing record in distilled database 230 to reflect the new information or content as would be apparent.
  • While this invention has been described in a preferred embodiment, other embodiments and variations are within the scope of the following claims. For example, formatting process 300 may format data using different radices or other character sets, and may use various data structures. The data structures may represent multiple fields, and depending on the application, will represent a variety of fields. For example, in a credit application, fields may include an account status, an account number, and a legal status, in addition to personal information about the account holder. In a medical diagnostic application, fields may include various alleles or other genetic characteristics detected in tissue samples.

Claims (22)

1-24. (canceled)
25. A method for converting information from a raw database into a distilled database, the raw database including a plurality of records, each of the plurality of records including a data field, the data field including a plurality of data elements, the method comprising:
converting the plurality of data elements in at least one non-numeric data field of one of the plurality of records in the raw database to a numeric value;
forming a vector including said numeric value, said vector representative of said one of the plurality of records in the raw database;
comparing said vector with a distilled matrix to determine whether said vector is included in said distilled matrix, said comparing using said numeric value;
including said vector in said distilled matrix if said vector is not included in said distilled matrix; and
forming the distilled database using said distilled matrix.
26. The method of claim 25, wherein said converting the plurality of data elements in at least one non-numeric data field of one of the plurality of records comprises:
representing each of the plurality of data elements as a digit in a number system, said number system having a radix at least equal to a number of possible values of a data element in said non-numeric data field, said digits collectively forming said numeric value in said number system.
27. The method of claim 25, further comprising:
maintaining information with said vector indicative of its origin in the raw database.
28. The method of claim 25, further comprising:
including said vector in a reference database; and
identifying an appropriate position for said vector in said reference database.
29. The method of claim 28, wherein said identifying an appropriate position for said vector comprises locating another vector similar to said vector.
30. The method of claim 29, wherein said locating another vector similar to said vector comprises numerically comparing said vector with said another vector.
31. The method of claim 28, further comprising locating a first vector in said reference database that is similar to a second vector in said reference database.
32. The method of claim 31, wherein said locating a first vector comprises locating said first vector in said reference database that is identifiable as said second vector in said reference database.
33. The method of claim 32, wherein said locating said first vector comprises locating said first vector in said reference database that is a duplicate of said second vector in said reference database.
34. The method of claim 31, further comprising forming a distilled vector from said first vector and said second vector that includes the best information from said first vector and said second vector.
35. The method of claim 34, wherein said comparing said vector with a distilled matrix comprises comparing said distilled vector with said distilled matrix to determine whether said distilled vector is included in said distilled matrix.
36. The method of claim 28, further comprising locating a first vector in said reference database that is dissimilar to every other vector in said reference database.
37. The method of claim 36, further comprising forming a distilled vector from said first vector.
38. The method of claim 37, wherein said comparing said vector with a distilled matrix comprises comparing said distilled vector with said distilled matrix to determine whether said distilled vector is included in said distilled matrix.
39-44. (canceled)
45. A method for generating a distilled database from a plurality of records, each of the plurality of records including at least one non-numeric data field, the non-numeric data field including a plurality of data elements, the method comprising:
determining a numeric value for the non-numeric data field from one of the plurality of records, said numeric value having a representation in a number system having a radix greater than or equal to a number of possible values of each of the plurality of data elements in the non-numeric data field;
forming a vector from said one of the plurality of records, said vector including said numeric value; and
comparing said vector with a vector in the distilled data base.
46. The method of claim 45, where said determining a numeric value from the non-numeric data field comprises determining a single numeric value for the non-numeric data field.
47. The method of claim 45, where said forming a vector from said one of the plurality of records comprises forming a vector from said one of the plurality of records, said vector including said numeric value instead of the non-numeric field.
48. A method for generating a database from a plurality of records, each of the plurality of records including at least one non-numeric data field, the non-numeric data field including a plurality of data elements, the plurality of data elements representing informational content, the method comprising:
representing the content represented by the plurality of data elements of the non-numeric data field as a numeric value;
forming a vector from said one of the plurality of records, said vector including said numeric value; and
including said vector in the database.
49. The method of claim 48, wherein said representing the content represented by the plurality of data elements of the non-numeric data field as a numeric value, said numeric value maintaining semantic significance of the non-numeric data field.
50. The method of claim 48, wherein said representing the content represented by the plurality of data elements of the non-numeric data field as a numeric value, said numeric value included in a number system having a radix greater than or equal to a number of possible values each of the plurality of data elements.
US12/758,180 1999-07-20 2010-04-12 System and Method for Organizing Data Abandoned US20110010398A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/758,180 US20110010398A1 (en) 1999-07-20 2010-04-12 System and Method for Organizing Data

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US09/357,301 US6424969B1 (en) 1999-07-20 1999-07-20 System and method for organizing data
US10/198,935 US7698283B2 (en) 1999-07-20 2002-07-22 System and method for organizing data
US12/758,180 US20110010398A1 (en) 1999-07-20 2010-04-12 System and Method for Organizing Data

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/198,935 Continuation US7698283B2 (en) 1999-07-20 2002-07-22 System and method for organizing data

Publications (1)

Publication Number Publication Date
US20110010398A1 true US20110010398A1 (en) 2011-01-13

Family

ID=23405051

Family Applications (8)

Application Number Title Priority Date Filing Date
US09/357,301 Expired - Lifetime US6424969B1 (en) 1999-07-20 1999-07-20 System and method for organizing data
US09/412,970 Expired - Lifetime US6457006B1 (en) 1999-07-20 1999-10-06 System and method for organizing data
US10/146,954 Expired - Lifetime US6751610B2 (en) 1999-07-20 2002-05-17 System and method for organizing data
US10/198,935 Expired - Fee Related US7698283B2 (en) 1999-07-20 2002-07-22 System and method for organizing data
US10/219,658 Abandoned US20030037051A1 (en) 1999-07-20 2002-08-16 System and method for organizing data
US12/758,180 Abandoned US20110010398A1 (en) 1999-07-20 2010-04-12 System and Method for Organizing Data
US13/775,489 Abandoned US20130297568A1 (en) 1999-07-20 2013-02-25 System and Method for Organizing Data
US14/221,682 Abandoned US20140207788A1 (en) 1999-07-20 2014-03-21 System and method for organizing data

Family Applications Before (5)

Application Number Title Priority Date Filing Date
US09/357,301 Expired - Lifetime US6424969B1 (en) 1999-07-20 1999-07-20 System and method for organizing data
US09/412,970 Expired - Lifetime US6457006B1 (en) 1999-07-20 1999-10-06 System and method for organizing data
US10/146,954 Expired - Lifetime US6751610B2 (en) 1999-07-20 2002-05-17 System and method for organizing data
US10/198,935 Expired - Fee Related US7698283B2 (en) 1999-07-20 2002-07-22 System and method for organizing data
US10/219,658 Abandoned US20030037051A1 (en) 1999-07-20 2002-08-16 System and method for organizing data

Family Applications After (2)

Application Number Title Priority Date Filing Date
US13/775,489 Abandoned US20130297568A1 (en) 1999-07-20 2013-02-25 System and Method for Organizing Data
US14/221,682 Abandoned US20140207788A1 (en) 1999-07-20 2014-03-21 System and method for organizing data

Country Status (5)

Country Link
US (8) US6424969B1 (en)
JP (2) JP2010092490A (en)
AU (1) AU2008203532A1 (en)
IL (1) IL147736A (en)
RU (1) RU2268488C2 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140316313A1 (en) * 2011-12-02 2014-10-23 Avex, Llc Spring-driven foot compression system
US20160186749A1 (en) * 2013-08-02 2016-06-30 Lufkin Industries, Llc Improved stator assembly for progressive cavity pumping systems
US9600808B1 (en) 2011-06-24 2017-03-21 Epic One Texas, Llc Secure payment card, method and system
US10369075B2 (en) 2015-03-03 2019-08-06 Avex, Llc Insole foot compression system and methods

Families Citing this family (97)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4046893B2 (en) * 1999-05-28 2008-02-13 富士通株式会社 Database copying apparatus, database copying method, and computer-readable recording medium recording a database copying program
US6829743B1 (en) * 1999-09-28 2004-12-07 Kabushiki Kaisha Toshiba Method of acquiring environment consideration condition information
US20020013551A1 (en) * 2000-05-26 2002-01-31 Akinori Zaitsu Medical pump monitoring system
US7584125B2 (en) * 2000-06-26 2009-09-01 Jpmorgan Chase Bank, N.A. Electronic check presentment system and method having an item sequence capability
US20020072953A1 (en) * 2000-12-08 2002-06-13 Michlowitz Eric S. Process, a method, a system and software architecture for evaluating supplier performance
CA2387277C (en) * 2001-05-25 2015-03-03 Hitachi, Ltd. Information processing system using nucleotide sequence-related information
US8294552B2 (en) * 2001-07-10 2012-10-23 Xatra Fund Mx, Llc Facial scan biometrics on a payment device
US20030050830A1 (en) * 2001-09-13 2003-03-13 William Troyer Method and apparatus for evaluating relative performance of a business in an association of the same or similar businesses
US7680817B2 (en) * 2001-10-15 2010-03-16 Maya-Systems Inc. Multi-dimensional locating system and method
US7606819B2 (en) 2001-10-15 2009-10-20 Maya-Systems Inc. Multi-dimensional locating system and method
US7092956B2 (en) * 2001-11-02 2006-08-15 General Electric Capital Corporation Deduplication system
US7624067B2 (en) * 2001-12-21 2009-11-24 Glynntech, Inc. Bankruptcy creditor manager internet system
US7334003B2 (en) * 2002-01-11 2008-02-19 First Data Corporation Methods and systems for extracting related information from flat files
US6961721B2 (en) * 2002-06-28 2005-11-01 Microsoft Corporation Detecting duplicate records in database
GB0217201D0 (en) * 2002-07-24 2002-09-04 Beach Solutions Ltd XML database differencing engine
US8131471B2 (en) * 2002-08-08 2012-03-06 Agilent Technologies, Inc. Methods and system for simultaneous visualization and manipulation of multiple data types
US20080058106A1 (en) 2002-10-07 2008-03-06 Maya-Systems Inc. Multi-dimensional locating game system and method
US20040158816A1 (en) * 2003-01-07 2004-08-12 Pandipati Radha K.C. Personal data organizer software
US8166033B2 (en) * 2003-02-27 2012-04-24 Parity Computing, Inc. System and method for matching and assembling records
US7225194B2 (en) * 2003-10-24 2007-05-29 Sap Ag Composite record identifier generator
US7739309B2 (en) 2004-09-30 2010-06-15 Microsoft Corporation Method, system, and computer-readable medium for merging data from multiple data sources for use in an electronic document
US7293022B2 (en) * 2005-04-14 2007-11-06 International Business Machines Corporation List update employing neutral sort keys
US8630996B2 (en) * 2005-05-05 2014-01-14 At&T Intellectual Property I, L.P. Identifying duplicate entries in a historical database
US20060271582A1 (en) * 2005-05-25 2006-11-30 Caterpillar Inc. System and method for analyzing raw data files
US8645313B1 (en) * 2005-05-27 2014-02-04 Microstrategy, Inc. Systems and methods for enhanced SQL indices for duplicate row entries
US7788651B2 (en) * 2005-09-02 2010-08-31 Microsoft Corporation Anonymous types
US7590624B2 (en) * 2005-09-12 2009-09-15 Lsi Corporation Process for identifying duplicate values in very large data sets
US8346725B2 (en) * 2006-09-15 2013-01-01 Oracle International Corporation Evolution of XML schemas involving partial data copy
US7870163B2 (en) * 2006-09-28 2011-01-11 Oracle International Corporation Implementation of backward compatible XML schema evolution in a relational database system
US20080120319A1 (en) 2006-11-21 2008-05-22 International Business Machines Corporation System and method for identifying computer users having files with common attributes
US7574446B2 (en) * 2006-12-06 2009-08-11 Catalyst Repository Systems, Inc. Converting arbitrary strings into numeric representations to facilitate complex comparisons
US8069404B2 (en) 2007-08-22 2011-11-29 Maya-Systems Inc. Method of managing expected documents and system providing same
US8601392B2 (en) 2007-08-22 2013-12-03 9224-5489 Quebec Inc. Timeline for presenting information
US20090150451A1 (en) * 2007-12-07 2009-06-11 Roche Diagnostics Operations, Inc. Method and system for selective merging of patient data
US8739050B2 (en) 2008-03-07 2014-05-27 9224-5489 Quebec Inc. Documents discrimination system and method thereof
CN101359330B (en) * 2008-05-04 2015-05-06 索意互动(北京)信息技术有限公司 Content extending method and system
US8607155B2 (en) 2008-09-12 2013-12-10 9224-5489 Quebec Inc. Method of managing groups of arrays of documents
EP2259210A2 (en) * 2009-06-05 2010-12-08 Siemens Aktiengesellschaft Method and device for analysing a database
US8719240B2 (en) * 2009-06-19 2014-05-06 International Business Machines Corporation Apparatus and method to sequentially deduplicate groups of files comprising the same file name but different file version numbers
RU2575987C2 (en) * 2010-02-11 2016-02-27 Телефонактиеболагет Л М Эрикссон (Пабл) Data management in directory database
US8650209B1 (en) * 2010-10-27 2014-02-11 Amdocs Software Systems Limited System, method, and computer program for determining most of the non duplicate records in high performance environments in an economical and fault-tolerant manner
US9122374B2 (en) 2011-02-01 2015-09-01 9224-5489 Quebec Inc. Expandable and collapsible arrays of documents
US9020830B2 (en) 2011-03-08 2015-04-28 Apptio, Inc. Hierarchy based dependent object relationships
CN102841906B (en) * 2011-06-24 2016-12-07 阿里巴巴集团控股有限公司 The transaction processing system of a kind of integration and transaction processing method
US10289657B2 (en) 2011-09-25 2019-05-14 9224-5489 Quebec Inc. Method of retrieving information elements on an undisplayed portion of an axis of information elements
US9275050B2 (en) * 2011-10-24 2016-03-01 Apptio, Inc. Global dictionaries using universal primitives
US9519693B2 (en) 2012-06-11 2016-12-13 9224-5489 Quebec Inc. Method and apparatus for displaying data element axes
US9646080B2 (en) 2012-06-12 2017-05-09 9224-5489 Quebec Inc. Multi-functions axis-based interface
US9460200B2 (en) 2012-07-02 2016-10-04 International Business Machines Corporation Activity recommendation based on a context-based electronic files search
US9262499B2 (en) 2012-08-08 2016-02-16 International Business Machines Corporation Context-based graphical database
US9619580B2 (en) 2012-09-11 2017-04-11 International Business Machines Corporation Generation of synthetic context objects
US8620958B1 (en) 2012-09-11 2013-12-31 International Business Machines Corporation Dimensionally constrained synthetic context objects database
US9251237B2 (en) 2012-09-11 2016-02-02 International Business Machines Corporation User-specific synthetic context object matching
US9223846B2 (en) 2012-09-18 2015-12-29 International Business Machines Corporation Context-based navigation through a database
US9741138B2 (en) 2012-10-10 2017-08-22 International Business Machines Corporation Node cluster relationships in a graph database
US20140136295A1 (en) 2012-11-13 2014-05-15 Apptio, Inc. Dynamic recommendations taken over time for reservations of information technology resources
US8931109B2 (en) 2012-11-19 2015-01-06 International Business Machines Corporation Context-based security screening for accessing data
US9229932B2 (en) 2013-01-02 2016-01-05 International Business Machines Corporation Conformed dimensional data gravity wells
US8983981B2 (en) 2013-01-02 2015-03-17 International Business Machines Corporation Conformed dimensional and context-based data gravity wells
US9069752B2 (en) 2013-01-31 2015-06-30 International Business Machines Corporation Measuring and displaying facets in context-based conformed dimensional data gravity wells
US9053102B2 (en) 2013-01-31 2015-06-09 International Business Machines Corporation Generation of synthetic context frameworks for dimensionally constrained hierarchical synthetic context-based objects
US9292506B2 (en) 2013-02-28 2016-03-22 International Business Machines Corporation Dynamic generation of demonstrative aids for a meeting
US10152526B2 (en) 2013-04-11 2018-12-11 International Business Machines Corporation Generation of synthetic context objects using bounded context objects
US9348794B2 (en) 2013-05-17 2016-05-24 International Business Machines Corporation Population of context-based data gravity wells
US9195608B2 (en) * 2013-05-17 2015-11-24 International Business Machines Corporation Stored data analysis
US10417591B2 (en) 2013-07-03 2019-09-17 Apptio, Inc. Recursive processing of object allocation rules
US9760620B2 (en) * 2013-07-23 2017-09-12 Salesforce.Com, Inc. Confidently adding snippets of search results to clusters of objects
US10325232B2 (en) 2013-09-20 2019-06-18 Apptio, Inc. Allocating heritage information in data models
US11244364B2 (en) 2014-02-13 2022-02-08 Apptio, Inc. Unified modeling of technology towers
RU2556425C1 (en) * 2014-02-14 2015-07-10 Закрытое акционерное общество "Эвентос" (ЗАО "Эвентос") Method for automatic iterative clusterisation of electronic documents according to semantic similarity, method for search in plurality of documents clustered according to semantic similarity and computer-readable media
US10380253B2 (en) * 2014-03-04 2019-08-13 International Business Machines Corporation Natural language processing with dynamic pipelines
US10387389B2 (en) 2014-09-30 2019-08-20 International Business Machines Corporation Data de-duplication
US9785698B2 (en) * 2014-12-31 2017-10-10 Business Objects Software Ltd. Behaviorally consistent cluster-wide data wrangling based on locally processed sampled data
US20160267172A1 (en) * 2015-03-09 2016-09-15 DonRiver, Inc. Constraint free model
US20160335294A1 (en) * 2015-05-15 2016-11-17 Bjorn J. Gruenwald System and Method for Organizing Data
US9350561B1 (en) 2015-05-27 2016-05-24 Apptio, Inc. Visualizing the flow of resources in an allocation model
GB2556504A (en) 2015-06-30 2018-05-30 Apptio Inc Infrastructure benchmarking based on dynamic cost modeling
US10089358B2 (en) 2015-09-21 2018-10-02 The Nielsen Company (Us), Llc Methods and apparatus to partition data
US10268979B2 (en) 2015-09-28 2019-04-23 Apptio, Inc. Intermediate resource allocation tracking in data models
US10387815B2 (en) 2015-09-29 2019-08-20 Apptio, Inc. Continuously variable resolution of resource allocation
US9384511B1 (en) 2015-12-16 2016-07-05 Apptio, Inc. Version control for resource allocation modeling
WO2017106773A1 (en) * 2015-12-19 2017-06-22 Von Drakk Viktor Method and device for correlating multiple tables in a database environment
US9529863B1 (en) 2015-12-21 2016-12-27 Apptio, Inc. Normalizing ingested data sets based on fuzzy comparisons to known data sets
US10726367B2 (en) 2015-12-28 2020-07-28 Apptio, Inc. Resource allocation forecasting
US10474974B2 (en) 2016-09-08 2019-11-12 Apptio, Inc. Reciprocal models for resource allocation
US10936978B2 (en) 2016-09-20 2021-03-02 Apptio, Inc. Models for visualizing resource allocation
US10482407B2 (en) 2016-11-14 2019-11-19 Apptio, Inc. Identifying resource allocation discrepancies
US10157356B2 (en) 2016-12-14 2018-12-18 Apptio, Inc. Activity based resource allocation modeling
US10671266B2 (en) 2017-06-05 2020-06-02 9224-5489 Quebec Inc. Method and apparatus of aligning information element axes
RU2667608C1 (en) * 2017-08-14 2018-09-21 Иван Александрович Баранов Method of ensuring the integrity of data
US10268980B1 (en) 2017-12-29 2019-04-23 Apptio, Inc. Report generation based on user responsibility
US11775552B2 (en) 2017-12-29 2023-10-03 Apptio, Inc. Binding annotations to data objects
US10324951B1 (en) 2017-12-29 2019-06-18 Apptio, Inc. Tracking and viewing model changes based on time
EP3785133A4 (en) 2018-04-24 2022-01-19 Von Drakk, Viktor Improved method and device for correlating multiple tables in a database environment
US11354093B1 (en) * 2020-10-19 2022-06-07 Khalid Omar Thabit Integer and characters prefix based methodologies combined with parallel data sort methodology enhance the execution performance of any string sorting algorithm
US11620271B2 (en) * 2021-08-11 2023-04-04 Sap Se Relationship analysis using vector representations of database tables
US20230222011A1 (en) * 2022-01-11 2023-07-13 Hiredscore Inc. System and method for mapping data from one organization's computer application to another organization's computer application

Citations (95)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3775753A (en) * 1971-01-04 1973-11-27 Texas Instruments Inc Vector order computing system
US4068300A (en) * 1973-12-13 1978-01-10 Honeywell Information Systems, Inc. Data processing system utilizing data field descriptors for processing data files
US4281391A (en) * 1979-01-15 1981-07-28 Leland Stanford Junior University Number theoretic processor
US4414629A (en) * 1979-04-02 1983-11-08 Waite John H Method and apparatus for making correlations and predictions using a finite field of data of unorganized and/or partially structured elements
US4635199A (en) * 1983-04-28 1987-01-06 Nec Corporation Pivot-type machine translating system comprising a pragmatic table for checking semantic structures, a pivot representation, and a result of translation
US4779192A (en) * 1983-12-23 1988-10-18 Hitachi, Ltd. Vector processor with a synchronously controlled operand fetch circuits
US4780810A (en) * 1984-05-25 1988-10-25 Hitachi, Ltd. Data processor with associative memory storing vector elements for vector conversion
US4864503A (en) * 1987-02-05 1989-09-05 Toltran, Ltd. Method of using a created international language as an intermediate pathway in translation between two national languages
US5148541A (en) * 1987-09-28 1992-09-15 Northern Telecom Limited Multilingual database system including sorting data using a master universal sort order for all languages
US5212639A (en) * 1990-04-05 1993-05-18 Sampson Wesley C Method and electronic apparatus for the classification of combinatorial data for the summarization and/or tabulation thereof
US5226158A (en) * 1989-05-24 1993-07-06 International Business Machines Corporation Method and apparatus for maintaining referential integrity within a relational database
US5225981A (en) * 1986-10-03 1993-07-06 Ricoh Company, Ltd. Language analyzer for morphemically and syntactically analyzing natural languages by using block analysis and composite morphemes
US5241648A (en) * 1990-02-13 1993-08-31 International Business Machines Corporation Hybrid technique for joining tables
US5287494A (en) * 1990-10-18 1994-02-15 International Business Machines Corporation Sorting/merging tree for determining a next tournament champion in each cycle by simultaneously comparing records in a path of the previous tournament champion
US5303149A (en) * 1989-11-03 1994-04-12 Janigian Paul C System for eliminating duplicate entries from a mailing list
US5418947A (en) * 1992-12-23 1995-05-23 At&T Corp. Locating information in an unsorted database utilizing a B-tree
US5471612A (en) * 1994-03-03 1995-11-28 Borland International, Inc. Electronic spreadsheet system and methods for compiling a formula stored in a spreadsheet into native machine code for execution by a floating-point unit upon spreadsheet recalculation
US5481704A (en) * 1991-09-27 1996-01-02 Computer Concepts Corp. Indexing/compression scheme for supporting graphics and data selection
US5499359A (en) * 1994-01-18 1996-03-12 Borland International, Inc. Methods for improved referential integrity in a relational database management system
US5508538A (en) * 1993-04-19 1996-04-16 California Institute Of Technology Signal processing applications of massively parallel charge domain computing devices
US5535300A (en) * 1988-12-30 1996-07-09 At&T Corp. Perceptual coding of audio signals using entropy coding and/or multiple power spectra
US5542087A (en) * 1993-10-15 1996-07-30 Hewlett-Packard Company Linear hashing for distributed records
US5559940A (en) * 1990-12-14 1996-09-24 Hutson; William H. Method and system for real-time information analysis of textual material
US5603022A (en) * 1994-09-23 1997-02-11 The Regents Of The University Of Michigan Data compression system and method representing records as differences between sorted domain ordinals representing field values
US5615301A (en) * 1994-09-28 1997-03-25 Rivers; W. L. Automated language translation system
US5619709A (en) * 1993-09-20 1997-04-08 Hnc, Inc. System and method of context vector generation and retrieval
US5621908A (en) * 1992-10-21 1997-04-15 Fujitsu Limited Parallel sorting system to reduce the amount of communication between processing devices
US5668989A (en) * 1996-09-18 1997-09-16 Mao; Decao Two-digit hybrid radix year numbers for year 2000 and beyond
US5678043A (en) * 1994-09-23 1997-10-14 The Regents Of The University Of Michigan Data compression and encryption system and method representing records as differences between sorted domain ordinals that represent field values
US5680627A (en) * 1991-02-15 1997-10-21 Texas Instruments Incorporated Method and apparatus for character preprocessing which translates textual description into numeric form for input to a neural network
US5721899A (en) * 1994-11-16 1998-02-24 Fujitsu Limited Retrieval apparatus using compressed trie node and retrieval method thereof
US5751957A (en) * 1995-09-12 1998-05-12 Hitachi, Ltd. Communication service system employing translation rules for communicating data in different languages along a network
US5799303A (en) * 1994-06-28 1998-08-25 Fujitsu Limited Apparatus and method for sorting attributes-mixed character strings
US5802509A (en) * 1993-09-21 1998-09-01 Hitachi, Ltd. Rule generation system and method of generating rule
US5806057A (en) * 1994-11-04 1998-09-08 Optima Direct, Inc. System for managing database of communication recipients
US5809501A (en) * 1996-01-30 1998-09-15 Telefonaktiebolaget L M Ericsson (Publ) Method and system of database management in an asynchronous transfer mode (ATM) environment
US5819268A (en) * 1995-01-10 1998-10-06 International Business Machines Corporation Method and system for testing for equality/difference in multiple tables of a database
US5819251A (en) * 1996-02-06 1998-10-06 Oracle Corporation System and apparatus for storage retrieval and analysis of relational and non-relational data
US5822743A (en) * 1997-04-08 1998-10-13 1215627 Ontario Inc. Knowledge-based information retrieval system
US5841981A (en) * 1995-09-28 1998-11-24 Hitachi Software Engineering Co., Ltd. Network management system displaying static dependent relation information
US5877482A (en) * 1994-06-09 1999-03-02 Reilly; Chris Security system for EFT using magnetic strip cards
US5892470A (en) * 1997-01-08 1999-04-06 Microsoft Corporation Method and system for mnemonic encoding of numbers
US5895470A (en) * 1997-04-09 1999-04-20 Xerox Corporation System for categorizing documents in a linked collection of documents
US5924091A (en) * 1996-08-28 1999-07-13 Sybase, Inc. Database system with improved methods for radix sorting
US5926811A (en) * 1996-03-15 1999-07-20 Lexis-Nexis Statistical thesaurus, method of forming same, and use thereof in query expansion in automated text searching
US5963642A (en) * 1996-12-30 1999-10-05 Goldstein; Benjamin D. Method and apparatus for secure storage of data
US5970490A (en) * 1996-11-05 1999-10-19 Xerox Corporation Integration platform for heterogeneous databases
US5974407A (en) * 1997-09-29 1999-10-26 Sacks; Jerome E. Method and apparatus for implementing a hierarchical database management system (HDBMS) using a relational database management system (RDBMS) as the implementing apparatus
US5978788A (en) * 1997-04-14 1999-11-02 International Business Machines Corporation System and method for generating multi-representations of a data cube
US5983216A (en) * 1997-09-12 1999-11-09 Infoseek Corporation Performing automated document collection and selection by providing a meta-index with meta-index values indentifying corresponding document collections
US5987456A (en) * 1997-10-28 1999-11-16 University Of Masschusetts Image retrieval by syntactic characterization of appearance
US5991713A (en) * 1997-11-26 1999-11-23 International Business Machines Corp. Efficient method for compressing, storing, searching and transmitting natural language text
US6026397A (en) * 1996-05-22 2000-02-15 Electronic Data Systems Corporation Data analysis system and method
US6029195A (en) * 1994-11-29 2000-02-22 Herz; Frederick S. M. System for customized electronic identification of desirable objects
US6070164A (en) * 1998-05-09 2000-05-30 Information Systems Corporation Database method and apparatus using hierarchical bit vector index structure
US6073140A (en) * 1997-07-29 2000-06-06 Acxiom Corporation Method and system for the creation, enhancement and update of remote data using persistent keys
US6081811A (en) * 1996-02-08 2000-06-27 Telefonaktiebolaget Lm Ericsson Method of database conversion including data verification
US6108670A (en) * 1997-11-24 2000-08-22 International Business Machines Corporation Checking and enabling database updates with a dynamic, multi-modal, rule based system
US6106562A (en) * 1990-03-08 2000-08-22 Corning Incorporated Apparatus and methods for predicting physical and chemical properties of materials
US6122628A (en) * 1997-10-31 2000-09-19 International Business Machines Corporation Multidimensional data clustering and dimension reduction for indexing and searching
US6128386A (en) * 1994-11-09 2000-10-03 Channel One Communications, Inc. Multiple number base encoder/decoder using a corresponding exclusive or function
US6134541A (en) * 1997-10-31 2000-10-17 International Business Machines Corporation Searching multidimensional indexes using associated clustering and dimension reduction information
US6138116A (en) * 1996-08-01 2000-10-24 Canon Kabushiki Kaisha Method and apparatus for retrieving data
US6138109A (en) * 1997-12-23 2000-10-24 Caterpillar Inc. Neural network diagnostic classification of complex binary systems
US6175835B1 (en) * 1996-07-26 2001-01-16 Ori Software Development, Ltd. Layered index with a basic unbalanced partitioned index that allows a balanced structure of blocks
US6175828B1 (en) * 1997-02-28 2001-01-16 Sharp Kabushiki Kaisha Retrieval apparatus
US6185567B1 (en) * 1998-05-29 2001-02-06 The Trustees Of The University Of Pennsylvania Authenticated access to internet based research and data services
US6196466B1 (en) * 1998-06-09 2001-03-06 Symbol Technologies, Inc. Data compression method using multiple base number systems
US6202141B1 (en) * 1998-06-16 2001-03-13 International Business Machines Corporation Method and apparatus for performing vector operation using separate multiplication on odd and even data elements of source vectors
US6208993B1 (en) * 1996-07-26 2001-03-27 Ori Software Development Ltd. Method for organizing directories
US6212530B1 (en) * 1998-05-12 2001-04-03 Compaq Computer Corporation Method and apparatus based on relational database design techniques supporting modeling, analysis and automatic hypertext generation for structured document collections
US6212532B1 (en) * 1998-10-22 2001-04-03 International Business Machines Corporation Text categorization toolkit
US6233571B1 (en) * 1993-06-14 2001-05-15 Daniel Egger Method and apparatus for indexing, searching and displaying data
US6233546B1 (en) * 1998-11-19 2001-05-15 William E. Datig Method and system for machine translation using epistemic moments and stored dictionary entries
US6236974B1 (en) * 1997-08-08 2001-05-22 Parasoft Corporation Method and apparatus for automated selection and organization of products including menus
US6249784B1 (en) * 1999-05-19 2001-06-19 Nanogen, Inc. System and method for searching and processing databases comprising named annotated text strings
US6253188B1 (en) * 1996-09-20 2001-06-26 Thomson Newspapers, Inc. Automated interactive classified ad system for the internet
US6263334B1 (en) * 1998-11-11 2001-07-17 Microsoft Corporation Density-based indexing method for efficient execution of high dimensional nearest-neighbor queries on large databases
US6275789B1 (en) * 1998-12-18 2001-08-14 Leo Moser Method and apparatus for performing full bidirectional translation between a source language and a linked alternative language
US6289353B1 (en) * 1997-09-24 2001-09-11 Webmd Corporation Intelligent query system for automatically indexing in a database and automatically categorizing users
US6301579B1 (en) * 1998-10-20 2001-10-09 Silicon Graphics, Inc. Method, system, and computer program product for visualizing a data structure
US6349309B1 (en) * 1999-05-24 2002-02-19 International Business Machines Corporation System and method for detecting clusters of information with application to e-commerce
US20020022953A1 (en) * 2000-05-24 2002-02-21 Bertolus Phillip Andre Indexing and searching ideographic characters on the internet
US6356936B1 (en) * 1998-09-01 2002-03-12 Bigfix, Inc. Relevance clause for computed relevance messaging
US6370513B1 (en) * 1997-08-08 2002-04-09 Parasoft Corporation Method and apparatus for automated selection, organization, and recommendation of items
US6463404B1 (en) * 1997-08-08 2002-10-08 British Telecommunications Public Limited Company Translation
US6523172B1 (en) * 1998-12-17 2003-02-18 Evolutionary Technologies International, Inc. Parser translator system and method
US6578025B1 (en) * 1999-06-11 2003-06-10 Abuzz Technologies, Inc. Method and apparatus for distributing information to users
US20030163595A1 (en) * 2002-02-26 2003-08-28 John Ta Task manager - method of forwarding messages among task blocks
US6675166B2 (en) * 2000-02-09 2004-01-06 The John Hopkins University Integrated multidimensional database
US6754666B1 (en) * 1999-08-19 2004-06-22 A2I, Inc. Efficient storage and access in a database management system
US20050080612A1 (en) * 2003-09-26 2005-04-14 Chai David T. Spelling and encoding method for ideographic symbols
US6910183B2 (en) * 2000-03-08 2005-06-21 International Business Machines Corporation File tagging and automatic conversion of data or files
US7020647B1 (en) * 1999-06-18 2006-03-28 International Business Machines Corporation Utilize encoded vector indexing for database grouping
US7107206B1 (en) * 1999-11-17 2006-09-12 United Nations Language conversion system

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US617835A (en) * 1899-01-17 Frank patak
EP0079465A3 (en) 1981-11-13 1985-01-23 International Business Machines Corporation Method for storing and accessing a relational data base
US4792793A (en) * 1987-05-28 1988-12-20 Amdahl Corporation Converting numbers between binary and another base
JPH07104784B2 (en) 1990-04-03 1995-11-13 富士ゼロックス株式会社 Digital data processor
US5197001A (en) 1990-05-14 1993-03-23 International Business Machines Corp. Bill of material and project network processing
US5675815A (en) * 1992-11-09 1997-10-07 Ricoh Company, Ltd. Language conversion system and text creating system using such
US5533051A (en) 1993-03-12 1996-07-02 The James Group Method for data compression
JP3185167B2 (en) 1993-12-24 2001-07-09 株式会社日立製作所 Data processing system
AU2473895A (en) 1994-05-05 1995-11-29 William H. Hutson A method and system for real-time information analysis of textual material
US6154213A (en) 1997-05-30 2000-11-28 Rennison; Earl F. Immersive movement-based interaction with large complex information structures
AU757667C (en) 1998-01-23 2003-10-16 Emc Corporation Access to content addressable data over a network
US6493662B1 (en) * 1998-02-11 2002-12-10 International Business Machines Corporation Rule-based number parser
EP0943990A3 (en) * 1998-02-27 2004-12-22 Texas Instruments Incorporated Method and system of providing dynamic optimization information in a code interpretive runtime environment
US5985470A (en) * 1998-03-16 1999-11-16 General Electric Company Thermal/environmental barrier coating system for silicon-based materials
US6343375B1 (en) * 1998-04-24 2002-01-29 International Business Machines Corporation Method for optimizing array bounds checks in programs
US6185673B1 (en) * 1998-06-30 2001-02-06 Sun Microsystems, Inc. Apparatus and method for array bounds checking with a shadow register file
US6327574B1 (en) * 1998-07-07 2001-12-04 Encirq Corporation Hierarchical models of consumer attributes for targeting content in a privacy-preserving manner
JP3280332B2 (en) * 1998-07-10 2002-05-13 インターナショナル・ビジネス・マシーンズ・コーポレーション Method and apparatus for performing versioning on loop, method and apparatus for collecting information on array range check in basic block, method for changing information on array range check, method for optimizing array range check, array range Method of generating code for checking, unnecessary array range check removal method and apparatus, method of selecting array range check, method of changing array range check, collection method of array range check, and handling judgment of array range check Method
EP1208474A2 (en) 1999-07-20 2002-05-29 Inmentia Inc Method and system for organizing data
US6408383B1 (en) * 2000-05-04 2002-06-18 Sun Microsystems, Inc. Array access boundary check by executing BNDCHK instruction with comparison specifiers
US6695461B2 (en) * 2000-12-25 2004-02-24 Seiko Epson Corporation Lamp unit, projector, and fixing method of light source lamp and reflector
US20020165707A1 (en) 2001-02-26 2002-11-07 Call Charles G. Methods and apparatus for storing and processing natural language text data as a sequence of fixed length integers
US20030023958A1 (en) * 2001-07-17 2003-01-30 Patel Mukesh K. Intermediate language accelerator chip
WO2003049594A2 (en) * 2001-12-10 2003-06-19 The Board Of Trustees Of The University Of Arkansas Minimally invasive treatment for breast cancer
US20030135788A1 (en) * 2002-01-11 2003-07-17 Raj Prakash Profile feedback assisted null check removal
CA2428821C (en) * 2003-05-15 2009-03-17 Ibm Canada Limited - Ibm Canada Limitee Accessing a platform independent input method editor from an underlying operating system
US7447665B2 (en) 2004-05-10 2008-11-04 Kinetx, Inc. System and method of self-learning conceptual mapping to organize and interpret data

Patent Citations (99)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3775753A (en) * 1971-01-04 1973-11-27 Texas Instruments Inc Vector order computing system
US4068300A (en) * 1973-12-13 1978-01-10 Honeywell Information Systems, Inc. Data processing system utilizing data field descriptors for processing data files
US4281391A (en) * 1979-01-15 1981-07-28 Leland Stanford Junior University Number theoretic processor
US4414629A (en) * 1979-04-02 1983-11-08 Waite John H Method and apparatus for making correlations and predictions using a finite field of data of unorganized and/or partially structured elements
US4635199A (en) * 1983-04-28 1987-01-06 Nec Corporation Pivot-type machine translating system comprising a pragmatic table for checking semantic structures, a pivot representation, and a result of translation
US4779192A (en) * 1983-12-23 1988-10-18 Hitachi, Ltd. Vector processor with a synchronously controlled operand fetch circuits
US4780810A (en) * 1984-05-25 1988-10-25 Hitachi, Ltd. Data processor with associative memory storing vector elements for vector conversion
US5225981A (en) * 1986-10-03 1993-07-06 Ricoh Company, Ltd. Language analyzer for morphemically and syntactically analyzing natural languages by using block analysis and composite morphemes
US5490061A (en) * 1987-02-05 1996-02-06 Toltran, Ltd. Improved translation system utilizing a morphological stripping process to reduce words to their root configuration to produce reduction of database size
US4864503A (en) * 1987-02-05 1989-09-05 Toltran, Ltd. Method of using a created international language as an intermediate pathway in translation between two national languages
US5148541A (en) * 1987-09-28 1992-09-15 Northern Telecom Limited Multilingual database system including sorting data using a master universal sort order for all languages
US5535300A (en) * 1988-12-30 1996-07-09 At&T Corp. Perceptual coding of audio signals using entropy coding and/or multiple power spectra
US5226158A (en) * 1989-05-24 1993-07-06 International Business Machines Corporation Method and apparatus for maintaining referential integrity within a relational database
US5303149A (en) * 1989-11-03 1994-04-12 Janigian Paul C System for eliminating duplicate entries from a mailing list
US5241648A (en) * 1990-02-13 1993-08-31 International Business Machines Corporation Hybrid technique for joining tables
US6106562A (en) * 1990-03-08 2000-08-22 Corning Incorporated Apparatus and methods for predicting physical and chemical properties of materials
US5212639A (en) * 1990-04-05 1993-05-18 Sampson Wesley C Method and electronic apparatus for the classification of combinatorial data for the summarization and/or tabulation thereof
US5287494A (en) * 1990-10-18 1994-02-15 International Business Machines Corporation Sorting/merging tree for determining a next tournament champion in each cycle by simultaneously comparing records in a path of the previous tournament champion
US5559940A (en) * 1990-12-14 1996-09-24 Hutson; William H. Method and system for real-time information analysis of textual material
US5680627A (en) * 1991-02-15 1997-10-21 Texas Instruments Incorporated Method and apparatus for character preprocessing which translates textual description into numeric form for input to a neural network
US5481704A (en) * 1991-09-27 1996-01-02 Computer Concepts Corp. Indexing/compression scheme for supporting graphics and data selection
US5621908A (en) * 1992-10-21 1997-04-15 Fujitsu Limited Parallel sorting system to reduce the amount of communication between processing devices
US5418947A (en) * 1992-12-23 1995-05-23 At&T Corp. Locating information in an unsorted database utilizing a B-tree
US5508538A (en) * 1993-04-19 1996-04-16 California Institute Of Technology Signal processing applications of massively parallel charge domain computing devices
US6233571B1 (en) * 1993-06-14 2001-05-15 Daniel Egger Method and apparatus for indexing, searching and displaying data
US5619709A (en) * 1993-09-20 1997-04-08 Hnc, Inc. System and method of context vector generation and retrieval
US5794178A (en) * 1993-09-20 1998-08-11 Hnc Software, Inc. Visualization of information using graphical representations of context vector based relationships and attributes
US5802509A (en) * 1993-09-21 1998-09-01 Hitachi, Ltd. Rule generation system and method of generating rule
US5542087A (en) * 1993-10-15 1996-07-30 Hewlett-Packard Company Linear hashing for distributed records
US5745896A (en) * 1994-01-18 1998-04-28 Borland Int Inc Referential integrity in a relational database management system
US5499359A (en) * 1994-01-18 1996-03-12 Borland International, Inc. Methods for improved referential integrity in a relational database management system
US5633998A (en) * 1994-03-03 1997-05-27 Schlafly; Roger Methods for compiling formulas stored in an electronic spreadsheet system
US5471612A (en) * 1994-03-03 1995-11-28 Borland International, Inc. Electronic spreadsheet system and methods for compiling a formula stored in a spreadsheet into native machine code for execution by a floating-point unit upon spreadsheet recalculation
US5877482A (en) * 1994-06-09 1999-03-02 Reilly; Chris Security system for EFT using magnetic strip cards
US5799303A (en) * 1994-06-28 1998-08-25 Fujitsu Limited Apparatus and method for sorting attributes-mixed character strings
US5603022A (en) * 1994-09-23 1997-02-11 The Regents Of The University Of Michigan Data compression system and method representing records as differences between sorted domain ordinals representing field values
US5678043A (en) * 1994-09-23 1997-10-14 The Regents Of The University Of Michigan Data compression and encryption system and method representing records as differences between sorted domain ordinals that represent field values
US5615301A (en) * 1994-09-28 1997-03-25 Rivers; W. L. Automated language translation system
US5806057A (en) * 1994-11-04 1998-09-08 Optima Direct, Inc. System for managing database of communication recipients
US6128386A (en) * 1994-11-09 2000-10-03 Channel One Communications, Inc. Multiple number base encoder/decoder using a corresponding exclusive or function
US5721899A (en) * 1994-11-16 1998-02-24 Fujitsu Limited Retrieval apparatus using compressed trie node and retrieval method thereof
US6029195A (en) * 1994-11-29 2000-02-22 Herz; Frederick S. M. System for customized electronic identification of desirable objects
US5819268A (en) * 1995-01-10 1998-10-06 International Business Machines Corporation Method and system for testing for equality/difference in multiple tables of a database
US5751957A (en) * 1995-09-12 1998-05-12 Hitachi, Ltd. Communication service system employing translation rules for communicating data in different languages along a network
US5841981A (en) * 1995-09-28 1998-11-24 Hitachi Software Engineering Co., Ltd. Network management system displaying static dependent relation information
US5809501A (en) * 1996-01-30 1998-09-15 Telefonaktiebolaget L M Ericsson (Publ) Method and system of database management in an asynchronous transfer mode (ATM) environment
US5819251A (en) * 1996-02-06 1998-10-06 Oracle Corporation System and apparatus for storage retrieval and analysis of relational and non-relational data
US6081811A (en) * 1996-02-08 2000-06-27 Telefonaktiebolaget Lm Ericsson Method of database conversion including data verification
US5926811A (en) * 1996-03-15 1999-07-20 Lexis-Nexis Statistical thesaurus, method of forming same, and use thereof in query expansion in automated text searching
US6026397A (en) * 1996-05-22 2000-02-15 Electronic Data Systems Corporation Data analysis system and method
US6208993B1 (en) * 1996-07-26 2001-03-27 Ori Software Development Ltd. Method for organizing directories
US6175835B1 (en) * 1996-07-26 2001-01-16 Ori Software Development, Ltd. Layered index with a basic unbalanced partitioned index that allows a balanced structure of blocks
US6138116A (en) * 1996-08-01 2000-10-24 Canon Kabushiki Kaisha Method and apparatus for retrieving data
US5924091A (en) * 1996-08-28 1999-07-13 Sybase, Inc. Database system with improved methods for radix sorting
US5668989A (en) * 1996-09-18 1997-09-16 Mao; Decao Two-digit hybrid radix year numbers for year 2000 and beyond
US6253188B1 (en) * 1996-09-20 2001-06-26 Thomson Newspapers, Inc. Automated interactive classified ad system for the internet
US5970490A (en) * 1996-11-05 1999-10-19 Xerox Corporation Integration platform for heterogeneous databases
US5963642A (en) * 1996-12-30 1999-10-05 Goldstein; Benjamin D. Method and apparatus for secure storage of data
US5892470A (en) * 1997-01-08 1999-04-06 Microsoft Corporation Method and system for mnemonic encoding of numbers
US6175828B1 (en) * 1997-02-28 2001-01-16 Sharp Kabushiki Kaisha Retrieval apparatus
US5822743A (en) * 1997-04-08 1998-10-13 1215627 Ontario Inc. Knowledge-based information retrieval system
US5895470A (en) * 1997-04-09 1999-04-20 Xerox Corporation System for categorizing documents in a linked collection of documents
US5978788A (en) * 1997-04-14 1999-11-02 International Business Machines Corporation System and method for generating multi-representations of a data cube
US6073140A (en) * 1997-07-29 2000-06-06 Acxiom Corporation Method and system for the creation, enhancement and update of remote data using persistent keys
US6370513B1 (en) * 1997-08-08 2002-04-09 Parasoft Corporation Method and apparatus for automated selection, organization, and recommendation of items
US6236974B1 (en) * 1997-08-08 2001-05-22 Parasoft Corporation Method and apparatus for automated selection and organization of products including menus
US6463404B1 (en) * 1997-08-08 2002-10-08 British Telecommunications Public Limited Company Translation
US5983216A (en) * 1997-09-12 1999-11-09 Infoseek Corporation Performing automated document collection and selection by providing a meta-index with meta-index values indentifying corresponding document collections
US6289353B1 (en) * 1997-09-24 2001-09-11 Webmd Corporation Intelligent query system for automatically indexing in a database and automatically categorizing users
US5974407A (en) * 1997-09-29 1999-10-26 Sacks; Jerome E. Method and apparatus for implementing a hierarchical database management system (HDBMS) using a relational database management system (RDBMS) as the implementing apparatus
US5987456A (en) * 1997-10-28 1999-11-16 University Of Masschusetts Image retrieval by syntactic characterization of appearance
US6134541A (en) * 1997-10-31 2000-10-17 International Business Machines Corporation Searching multidimensional indexes using associated clustering and dimension reduction information
US6122628A (en) * 1997-10-31 2000-09-19 International Business Machines Corporation Multidimensional data clustering and dimension reduction for indexing and searching
US6108670A (en) * 1997-11-24 2000-08-22 International Business Machines Corporation Checking and enabling database updates with a dynamic, multi-modal, rule based system
US5991713A (en) * 1997-11-26 1999-11-23 International Business Machines Corp. Efficient method for compressing, storing, searching and transmitting natural language text
US6138109A (en) * 1997-12-23 2000-10-24 Caterpillar Inc. Neural network diagnostic classification of complex binary systems
US6070164A (en) * 1998-05-09 2000-05-30 Information Systems Corporation Database method and apparatus using hierarchical bit vector index structure
US6212530B1 (en) * 1998-05-12 2001-04-03 Compaq Computer Corporation Method and apparatus based on relational database design techniques supporting modeling, analysis and automatic hypertext generation for structured document collections
US6185567B1 (en) * 1998-05-29 2001-02-06 The Trustees Of The University Of Pennsylvania Authenticated access to internet based research and data services
US6196466B1 (en) * 1998-06-09 2001-03-06 Symbol Technologies, Inc. Data compression method using multiple base number systems
US6202141B1 (en) * 1998-06-16 2001-03-13 International Business Machines Corporation Method and apparatus for performing vector operation using separate multiplication on odd and even data elements of source vectors
US6356936B1 (en) * 1998-09-01 2002-03-12 Bigfix, Inc. Relevance clause for computed relevance messaging
US6301579B1 (en) * 1998-10-20 2001-10-09 Silicon Graphics, Inc. Method, system, and computer program product for visualizing a data structure
US6212532B1 (en) * 1998-10-22 2001-04-03 International Business Machines Corporation Text categorization toolkit
US6263334B1 (en) * 1998-11-11 2001-07-17 Microsoft Corporation Density-based indexing method for efficient execution of high dimensional nearest-neighbor queries on large databases
US6233546B1 (en) * 1998-11-19 2001-05-15 William E. Datig Method and system for machine translation using epistemic moments and stored dictionary entries
US6523172B1 (en) * 1998-12-17 2003-02-18 Evolutionary Technologies International, Inc. Parser translator system and method
US6275789B1 (en) * 1998-12-18 2001-08-14 Leo Moser Method and apparatus for performing full bidirectional translation between a source language and a linked alternative language
US6249784B1 (en) * 1999-05-19 2001-06-19 Nanogen, Inc. System and method for searching and processing databases comprising named annotated text strings
US6349309B1 (en) * 1999-05-24 2002-02-19 International Business Machines Corporation System and method for detecting clusters of information with application to e-commerce
US6578025B1 (en) * 1999-06-11 2003-06-10 Abuzz Technologies, Inc. Method and apparatus for distributing information to users
US7020647B1 (en) * 1999-06-18 2006-03-28 International Business Machines Corporation Utilize encoded vector indexing for database grouping
US6754666B1 (en) * 1999-08-19 2004-06-22 A2I, Inc. Efficient storage and access in a database management system
US7107206B1 (en) * 1999-11-17 2006-09-12 United Nations Language conversion system
US6675166B2 (en) * 2000-02-09 2004-01-06 The John Hopkins University Integrated multidimensional database
US6910183B2 (en) * 2000-03-08 2005-06-21 International Business Machines Corporation File tagging and automatic conversion of data or files
US20020022953A1 (en) * 2000-05-24 2002-02-21 Bertolus Phillip Andre Indexing and searching ideographic characters on the internet
US20030163595A1 (en) * 2002-02-26 2003-08-28 John Ta Task manager - method of forwarding messages among task blocks
US20050080612A1 (en) * 2003-09-26 2005-04-14 Chai David T. Spelling and encoding method for ideographic symbols

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9600808B1 (en) 2011-06-24 2017-03-21 Epic One Texas, Llc Secure payment card, method and system
US20140316313A1 (en) * 2011-12-02 2014-10-23 Avex, Llc Spring-driven foot compression system
US10799415B2 (en) * 2011-12-02 2020-10-13 Avex, Llc Spring-driven foot compression system
US20160186749A1 (en) * 2013-08-02 2016-06-30 Lufkin Industries, Llc Improved stator assembly for progressive cavity pumping systems
US10369075B2 (en) 2015-03-03 2019-08-06 Avex, Llc Insole foot compression system and methods

Also Published As

Publication number Publication date
US7698283B2 (en) 2010-04-13
AU2008203532A1 (en) 2008-08-28
US20030046285A1 (en) 2003-03-06
RU2268488C2 (en) 2006-01-20
US20130297568A1 (en) 2013-11-07
US6457006B1 (en) 2002-09-24
IL147736A (en) 2008-08-07
JP2010092490A (en) 2010-04-22
US20140207788A1 (en) 2014-07-24
US20020194176A1 (en) 2002-12-19
US6424969B1 (en) 2002-07-23
US20030037051A1 (en) 2003-02-20
US6751610B2 (en) 2004-06-15
JP2011175648A (en) 2011-09-08

Similar Documents

Publication Publication Date Title
US7698283B2 (en) System and method for organizing data
US6542896B1 (en) System and method for organizing data
US5237678A (en) System for storing and manipulating information in an information base
US7809730B2 (en) Method and system for creating database profiling indices
US7756804B2 (en) Automated model building and evaluation for data mining system
US5983222A (en) Method and apparatus for computing association rules for data mining in large database
US20040133581A1 (en) Database management system, data structure generating method for database management system, and storage medium therefor
US6741999B2 (en) String tree utility data structure and order n sort
US6553359B1 (en) Data mining for association rules and sequential patterns within data of inhomogeneous type
EP1043666A2 (en) A system for identification of selectively related database records
EP1116137B1 (en) Database, and methods of data storage and retrieval
US20030055838A1 (en) Data storing method and data storing structure
US6898530B1 (en) Method and apparatus for extracting attributes from sequence strings and biopolymer material
JP2002063363A (en) Insurance contract information management system and customer index generating method
Winarko et al. A signature-based indexing method for efficient content-based retrieval of relative temporal patterns
Abdalla et al. Leverage data quality improvement for big data analytics
JPH0997312A (en) Recognized character correction device and data base retrieval device
SE518744C2 (en) Data extraction method for Internet applications, involves replacing contents of other fields in respective data table in new record stored in predetermined table for expansion

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION