郭人通-Milvus：探索云原生的向量數據庫（GOTC上海會場）（19頁）.pdf

1、Milvus：探索云原生的向量數據庫郭人通 2021年07月10日合伙人&架構師計算機軟件與理論博士興趣領域：分布式系統、數據庫、異構計算CCF 分布式計算與系統專委會委員郭人通Milvus 系統架構師Why Vector DatabaseInt,float,string,textjsonimagevideoaudiodomain specific Structured dataUnstructured dataABCDEFGABCDEFG2021.04.102021.04.10Data are Increasing Horizontally:TypesWhy Vector DatabaseD

2、ata are Increasing Vertically:SemanticsembeddingsRicher semanticsembeddingsembeddingsWhy Vector DatabasebooleanembeddingIntfloatstringTypes with high abstractionWhy Vector Database80%data growth is unstructured,over 40,000 Exabytes per year The big pictureRaw Data(Structured/Unstructured Data)Latent

3、-semantic Data(Embedding Vectors)Search&Analytics EnginesVector EngineCo-processorsQuery Parser/PlannerAbout MilvusThe Idea2018.10Milvus 0.1Release2019.041st SeedUser2019.06OpenSource2019.102020.03Milvus 1.0Release2021.03Milvus 2.0 Release2021.062019.06ProjectGraduationJoinedLF AI&DATAA brief histor

4、yStone AgeBronze AgeIron AgeRoman TimesDatabaseFrameworksFor AI ModelFrameworks For AI DataMagnetic Taps1970sCODASYL IMSRelational Model1980sSQLSIGMODVLDBERIngresSystem RDBMSs for PCDB2OracleSybaseInformix1990sMySQLNowPostgresSQLSnowflakeHBaseCloud NativeCAP2010s2016NowMATLABOpenNNTorchCaffeGPU supp

5、ortChainerTheanoTensorflowCaffe2PytorchCNTKMXNetKerasONNX2019NowFaissAnnoyhnswlibDB-pluginES-pluginPineconeMilvusProximaWe are hereKey challengesFast System Evolution01Multi-environment Deployment02Hardware Cost03Diverse Workloads04Complex,Hybrid Query05Architecture:logical log as the system backbon

6、eArchitecture:take a closer lookArchitecture:take a closer lookCoordinatorsLog BrokerLog SubscribersStorageFrom Milvus 1.0 to Milvus 2.0MilvusMilvus FunctionalitiesProxy,DDL handling,DML handling,DQL handlingMetaLog Pub-subDataetcdPulsarRocksDBS3MinIOTxnKVKVMsgStreamFrom Milvus 1.0 to Milvus 2.0Milv

7、us FunctionalitiesProxy,DDL handling,DML handling,DQL handlingMetaLog Pub-subDataetcdPulsarRocksDBS3MinIOTxnKVKVMsgStreamMetaLogPub-subDataTxnKVKVMsgStreamProxy NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordFrom Milvus 1.0 to Milvus 2.0MetaLogPub-subDataTxnKVKVMsgStreamProxy N

8、odeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordMetaLogPub-subDataTxnKVKVMsgStreamProxy NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordFrom Milvus 1.0 to Milvus 2.0MetaLogPub-subDataTxnKVKVMsgStreamProxy NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordInd

9、exNodeRootCoordMetaLogPub-subDataTxnKVKVMsgStreamProxy NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordFrom Milvus 1.0 to Milvus 2.0MetaLogPub-subDataTxnKVKVMsgStreamProxy NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordMetaLogPub-subDataTxnKVKVMsgStreamProx

10、y NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordFrom Milvus 1.0 to Milvus 2.0MetaLogPub-subDataTxnKVKVMsgStreamProxy NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordMetaLogPub-subDataTxnKVKVMsgStreamProxy NodeQuery CoordQueryNodeDataCoordDataNodeIndexCoordIndexNodeRootCoordSidecarSidecarSidecarSidecarSidecarSidecarSidecarSidecark8s