What is the construction, real-time updating, and maintenance mechanism of Facebook's graph data? Please explain in detail the underlying technical implementation.

26 Tem 20:20

In building the internal knowledge graph for the enterprise I oversee, the initial construction process closely mirrored Facebook’s approach: we first consolidated user behavior logs (clicks, views, likes, comments, etc.) and user relationship data (follows, organizational structure, project collaborations) into a Kafka topic, then used Spark Streaming to batch-write these events into Neo4j/JanusGraph. Each edge creation followed a set of association rules—e.g., “User A liked Post B” would generate an (A)-[LIKE]->(B) edge in the graph, while simultaneously syncing the user’s interest tags to properties for downstream similarity calculations. For real-time updates, I implemented Flink + CDC (Change Data Capture) to directly monitor MySQL binlogs, instantly mapping changes to user profiles (names, privacy settings) and additions/deletions in relationships to vertices and edges in the graph database—ensuring the knowledge graph reflects the latest state within seconds. To maintain data consistency at scale, the graph is sharded across multiple partitions, each replicated via Paxos/Raft consensus protocols. Write conflicts are resolved using optimistic locking with version numbers. Privacy compliance is handled at the application layer via ACLs (Access Control Lists) and differential privacy noise injection: during queries, we first verify the requester’s permissions, returning only property fields compliant with privacy policies, and add Laplace noise to aggregated statistics. This balances GDPR compliance with recommendation accuracy. The entire pipeline enables sub-second updates and high availability even at the scale of hundreds of millions of nodes.