Milvus Metadata Management (2) Fields in the Metadata Table

12/27/2019

by Yihua Mo

Blog>Article

Milvus Metadata Management (2)

Fields in the Metadata Table

Author: Yihua Mo

Date: 2019-12-27

In the last blog, we mentioned how to view your metadata using MySQL or SQLite. This article mainly intends to introduce in detail the fields in the metadata tables.

Fields in the "Tables” table

Take SQLite as an example. The following result comes from 0.5.0. Some fields are added to 0.6.0, which will be introduced later. There is a row in Tables specifying a 512-dimensional vector table with the name table_1. When the table is created, index_file_size is 1024 MB, engine_type is 1 (FLAT), nlist is 16384, metric_type is 1 (Euclidean distance L2). id is the unique identifier of the table. state is the state of the table with 0 indicating a normal state. created_on is the creation time. flag is the flag reserved for internal use.

tables

The following table shows field types and descriptions of the fields in Tables.

Field Name Data Type Description
id int64 Unique identifier of the vector table. id automatically increments.
table_id string Name of the vector table. table_id must be user-defined and follow Linux filename guidelines.
state int32 State of the vector table. 0 stands for normal and 1 stands for deleted (soft delete).
dimension int16 Vector dimension of the vector table. Must be user-defined.
created_on int64 Number of milliseconds from Jan 1, 1970 to the time when the table is created.
flag int64 Flag for internal use, such as whether the vector id is user-defined. The default is 0.
index_file_size int64 If the size of a data file reaches index_file_size, the file is not combined and is used to build indexes. The default is 1024 (MB).
engine_type int32 Type of index to build for a vector table. The default is 0, which specifies invalid index. 1 specifies FLAT. 2 specifies IVFLAT. 3 specifies IVFSQ8. 4 specifies NSG. 5 specifies IVFSQ8H.
nlist int32 Number of clusters the vectors in each data file are divided into when the index is being built. The default is 16384.
metric_type int32 Method to compute vector distance. 1 specifies Euclidean distance (L1) and 2 specifies inner product.

Table partitioning is enabled in 0.6.0 with a few new fields, including owner_tablepartition_tag and version. A vector table, table_1, has a partition called table_1_p1, which is also a vector table. partition_name corresponds to table_id. Fields in a partition table are inherited from the owner table, with the owner table field specifying the name of the owner table and the partition_tag field specifying the tag of the partition.

tables_new

The following table shows new fields in 0.6.0:

Field Name Data Type Description
owner_table string Parent table of the partition.
partition_tag string Tag of the partition. Must not be an empty string.
version string Milvus version.

Fields in the “TableFiles" table

The following example contains two files, which both belong to the table_1 vector table. The index type (engine_type) of the first file is 1 (FLAT); file status (file_type) is 7 (backup of the original file); file_size is 411200113 bytes; number of vector rows is 200,000. The index type of the second file is 2 (IVFLAT); file status is 3 (index file). The second file is actually the index of the first file. We will introduce more information in upcoming articles.

tablefiles

The following table shows fields and descriptions of TableFiles:

Field Name Data Type Description
id int64 Unique identifier of a vector table. id automatically increments.
table_id string Name of the vector table.
engine_type int32 Type of index to build for a vector table. The default is 0, which specifies invalid index. 1 specifies FLAT. 2 specifies IVFLAT. 3 specifies IVFSQ8. 4 specifies NSG. 5 specifies IVFSQ8H.
file_id string Filename generated from file creation time. Equals 1000 multiplied by the number of milliseconds from Jan 1, 1970 to the time when the table is created.
file_type int32 File status. 0 specifies a newly generated raw vector data file. 1 specifies raw vector data file. 2 specifies that index will be built for the file. 3 specifies that the file is an index file. 4 specifies that the file will be deleted (soft delete). 5 specifies that the file is newly-generated and used to store combination data. 6 specifies that the file is newly-generated and used to store index data. 7 specifies the backup status of the raw vector data file.
file_size int64 File size in bytes.
row_count int64 Number of vectors in a file.
updated_time int64 Timestamp for the latest update time, which specifies the number of milliseconds from Jan 1, 1970 to the time when the table is created.
created_on int64 Number of milliseconds from Jan 1, 1970 to the time when the table is created.
date int32 Date when the table is created. It is still here for historical reasons and will be removed in future versions.