Milvus Metadata Management (2) Fields in the Metadata Table
Fields in the Metadata Table
Author: Yihua Mo
Date: 2019-12-27
In the last blog, we mentioned how to view your metadata using MySQL or SQLite. This article mainly intends to introduce in detail the fields in the metadata tables.
Fields in the "Tables” table
Take SQLite as an example. The following result comes from 0.5.0. Some fields are added to 0.6.0, which will be introduced later. There is a row in Tables specifying a 512-dimensional vector table with the name table_1. When the table is created, index_file_size is 1024 MB, engine_type is 1 (FLAT), nlist is 16384, metric_type is 1 (Euclidean distance L2). id is the unique identifier of the table. state is the state of the table with 0 indicating a normal state. created_on is the creation time. flag is the flag reserved for internal use.
tables
The following table shows field types and descriptions of the fields in Tables.
| Field Name | Data Type | Description |
|---|---|---|
id | int64 | Unique identifier of the vector table. id automatically increments. |
table_id | string | Name of the vector table. table_id must be user-defined and follow Linux filename guidelines. |
state | int32 | State of the vector table. 0 stands for normal and 1 stands for deleted (soft delete). |
dimension | int16 | Vector dimension of the vector table. Must be user-defined. |
created_on | int64 | Number of milliseconds from Jan 1, 1970 to the time when the table is created. |
flag | int64 | Flag for internal use, such as whether the vector id is user-defined. The default is 0. |
index_file_size | int64 | If the size of a data file reaches index_file_size, the file is not combined and is used to build indexes. The default is 1024 (MB). |
engine_type | int32 | Type of index to build for a vector table. The default is 0, which specifies invalid index. 1 specifies FLAT. 2 specifies IVFLAT. 3 specifies IVFSQ8. 4 specifies NSG. 5 specifies IVFSQ8H. |
nlist | int32 | Number of clusters the vectors in each data file are divided into when the index is being built. The default is 16384. |
metric_type | int32 | Method to compute vector distance. 1 specifies Euclidean distance (L1) and 2 specifies inner product. |
Table partitioning is enabled in 0.6.0 with a few new fields, including owner_table，partition_tag and version. A vector table, table_1, has a partition called table_1_p1, which is also a vector table. partition_name corresponds to table_id. Fields in a partition table are inherited from the owner table, with the owner table field specifying the name of the owner table and the partition_tag field specifying the tag of the partition.
tables_new
The following table shows new fields in 0.6.0:
| Field Name | Data Type | Description |
|---|---|---|
owner_table | string | Parent table of the partition. |
partition_tag | string | Tag of the partition. Must not be an empty string. |
version | string | Milvus version. |
Fields in the “TableFiles" table
The following example contains two files, which both belong to the table_1 vector table. The index type (engine_type) of the first file is 1 (FLAT); file status (file_type) is 7 (backup of the original file); file_size is 411200113 bytes; number of vector rows is 200,000. The index type of the second file is 2 (IVFLAT); file status is 3 (index file). The second file is actually the index of the first file. We will introduce more information in upcoming articles.
tablefiles
The following table shows fields and descriptions of TableFiles:
| Field Name | Data Type | Description |
|---|---|---|
id | int64 | Unique identifier of a vector table. id automatically increments. |
table_id | string | Name of the vector table. |
engine_type | int32 | Type of index to build for a vector table. The default is 0, which specifies invalid index. 1 specifies FLAT. 2 specifies IVFLAT. 3 specifies IVFSQ8. 4 specifies NSG. 5 specifies IVFSQ8H. |
file_id | string | Filename generated from file creation time. Equals 1000 multiplied by the number of milliseconds from Jan 1, 1970 to the time when the table is created. |
file_type | int32 | File status. 0 specifies a newly generated raw vector data file. 1 specifies raw vector data file. 2 specifies that index will be built for the file. 3 specifies that the file is an index file. 4 specifies that the file will be deleted (soft delete). 5 specifies that the file is newly-generated and used to store combination data. 6 specifies that the file is newly-generated and used to store index data. 7 specifies the backup status of the raw vector data file. |
file_size | int64 | File size in bytes. |
row_count | int64 | Number of vectors in a file. |
updated_time | int64 | Timestamp for the latest update time, which specifies the number of milliseconds from Jan 1, 1970 to the time when the table is created. |
created_on | int64 | Number of milliseconds from Jan 1, 1970 to the time when the table is created. |
date | int32 | Date when the table is created. It is still here for historical reasons and will be removed in future versions. |
Related blogs
Like the article? Spread the word
