milvus-logo
LFAI

December 27, 2019by Yihua Mo

Milvus Metadata Management (2) Fields in the Metadata Table

  • Engineering

Milvus Metadata Management (2)

Fields in the Metadata Table

Author: Yihua Mo

Date: 2019-12-27

In the last blog, we mentioned how to view your metadata using MySQL or SQLite. This article mainly intends to introduce in detail the fields in the metadata tables.

Fields in the "Tables” table

Take SQLite as an example. The following result comes from 0.5.0. Some fields are added to 0.6.0, which will be introduced later. There is a row in Tables specifying a 512-dimensional vector table with the name table_1. When the table is created, index_file_size is 1024 MB, engine_type is 1 (FLAT), nlist is 16384, metric_type is 1 (Euclidean distance L2). id is the unique identifier of the table. state is the state of the table with 0 indicating a normal state. created_on is the creation time. flag is the flag reserved for internal use.

tables tables

The following table shows field types and descriptions of the fields in Tables.

Field NameData TypeDescription
idint64Unique identifier of the vector table. id automatically increments.
table_idstringName of the vector table. table_id must be user-defined and follow Linux filename guidelines.
stateint32State of the vector table. 0 stands for normal and 1 stands for deleted (soft delete).
dimensionint16Vector dimension of the vector table. Must be user-defined.
created_onint64Number of milliseconds from Jan 1, 1970 to the time when the table is created.
flagint64Flag for internal use, such as whether the vector id is user-defined. The default is 0.
index_file_sizeint64If the size of a data file reaches index_file_size, the file is not combined and is used to build indexes. The default is 1024 (MB).
engine_typeint32Type of index to build for a vector table. The default is 0, which specifies invalid index. 1 specifies FLAT. 2 specifies IVFLAT. 3 specifies IVFSQ8. 4 specifies NSG. 5 specifies IVFSQ8H.
nlistint32Number of clusters the vectors in each data file are divided into when the index is being built. The default is 16384.
metric_typeint32Method to compute vector distance. 1 specifies Euclidean distance (L1) and 2 specifies inner product.

Table partitioning is enabled in 0.6.0 with a few new fields, including owner_tablepartition_tag and version. A vector table, table_1, has a partition called table_1_p1, which is also a vector table. partition_name corresponds to table_id. Fields in a partition table are inherited from the owner table, with the owner table field specifying the name of the owner table and the partition_tag field specifying the tag of the partition.

tables_new tables_new

The following table shows new fields in 0.6.0:

Field NameData TypeDescription
owner_tablestringParent table of the partition.
partition_tagstringTag of the partition. Must not be an empty string.
versionstringMilvus version.

Fields in the “TableFiles" table

The following example contains two files, which both belong to the table_1 vector table. The index type (engine_type) of the first file is 1 (FLAT); file status (file_type) is 7 (backup of the original file); file_size is 411200113 bytes; number of vector rows is 200,000. The index type of the second file is 2 (IVFLAT); file status is 3 (index file). The second file is actually the index of the first file. We will introduce more information in upcoming articles.

tablefiles tablefiles

The following table shows fields and descriptions of TableFiles:

Field NameData TypeDescription
idint64Unique identifier of a vector table. id automatically increments.
table_idstringName of the vector table.
engine_typeint32Type of index to build for a vector table. The default is 0, which specifies invalid index. 1 specifies FLAT. 2 specifies IVFLAT. 3 specifies IVFSQ8. 4 specifies NSG. 5 specifies IVFSQ8H.
file_idstringFilename generated from file creation time. Equals 1000 multiplied by the number of milliseconds from Jan 1, 1970 to the time when the table is created.
file_typeint32File status. 0 specifies a newly generated raw vector data file. 1 specifies raw vector data file. 2 specifies that index will be built for the file. 3 specifies that the file is an index file. 4 specifies that the file will be deleted (soft delete). 5 specifies that the file is newly-generated and used to store combination data. 6 specifies that the file is newly-generated and used to store index data. 7 specifies the backup status of the raw vector data file.
file_sizeint64File size in bytes.
row_countint64Number of vectors in a file.
updated_timeint64Timestamp for the latest update time, which specifies the number of milliseconds from Jan 1, 1970 to the time when the table is created.
created_onint64Number of milliseconds from Jan 1, 1970 to the time when the table is created.
dateint32Date when the table is created. It is still here for historical reasons and will be removed in future versions.

Like the article? Spread the word

Keep Reading