使用 APOC NLP 进行实体提取
实体提取简介
实体提取从非结构化文本中提取并返回该文本中包含的命名实体列表。
AWS、GCP 和 Azure 分别提供了 NLP API,这些 API 由 apoc.nlp 过程封装。
导入文本文档
我们将从 卫报足球 RSS Feed 导入一些文本文档。我们可以使用 apoc.load.xml
来完成此操作,如下面的查询所示
CALL apoc.load.xml("https://www.theguardian.com/football/rss", "rss/channel/item") YIELD value WITH [child in value._children WHERE child._type = "title" | child._text][0] AS title, [child in value._children WHERE child._type = "link" | child._text][0] AS guid, apoc.text.regreplace( [child in value._children WHERE child._type = "description" | child._text][0], '<[^>]*>', ' ' ) AS body, [child in value._children WHERE child._type = "category" | child._text] AS categories MERGE (a:Article {id: guid}) SET a.body = body, a.title = title WITH a, categories CALL { WITH a, categories UNWIND categories AS category MERGE (c:Category {name: category}) MERGE (a)-[:IN_CATEGORY]->(c) RETURN count(*) AS categoryCount } RETURN a.id AS articleId, a.title AS title, categoryCount;
我们可以在下面的图表中看到导入图的 Neo4j 浏览器可视化效果
构建实体图
现在我们将构建我们的实体图。我们将结合使用 apoc.periodic.iterate
和特定于云的流式过程来
-
提取每篇文章的实体,并过滤特定实体类型
-
为每个实体创建一个节点
-
从每个实体到文章创建一个关系
我们可以使用以下查询来完成此操作
CALL apoc.periodic.iterate(
"MATCH (a:Article) WHERE not(exists(a.processed)) RETURN a",
"CALL apoc.nlp.aws.entities.stream([item in $_batch | item.a], {
key: $awsApiKey,
secret: $awsApiSecret,
nodeProperty: 'body'
})
YIELD node AS a, value
SET a.processed = true
WITH a, value
UNWIND value.entities AS entity
WITH a, entity
WHERE entity.type IN ['COMMERCIAL_ITEM', 'PERSON', 'ORGANIZATION', 'LOCATION', 'EVENT']
CALL apoc.merge.node(
['Entity', apoc.text.capitalize(toLower(entity.type))],
{name: entity.text}, {}, {}
)
YIELD node AS e
MERGE (a)-[entityRel:AWS_ENTITY]->(e)
SET entityRel.score = entity.score
RETURN count(*)",
{ batchMode: "BATCH_SINGLE",
batchSize: 25,
params: {awsApiKey: $awsApiKey, awsApiSecret: $awsApiSecret}}
)
YIELD batches, total, timeTaken, committedOperations, errorMessages, batch, operations
RETURN batches, total, timeTaken, committedOperations, errorMessages, batch, operations;
批次 | 总数 | 花费时间 | 已提交的操作 | 错误消息 | 批次 | 操作 |
---|---|---|---|---|---|---|
3 |
59 |
2 |
59 |
{} |
{total: 3, committed: 3, failed: 0, errors: {}} |
{total: 59, committed: 59, failed: 0, errors: {}} |
CALL apoc.periodic.iterate(
"MATCH (a:Article) WHERE not(exists(a.processed)) RETURN a",
"CALL apoc.nlp.gcp.entities.stream([item in $_batch | item.a], {
key: $gcpApiKey,
nodeProperty: 'body'
})
YIELD node AS a, value
SET a.processed = true
WITH a, value
UNWIND value.entities AS entity
WITH a, entity
WHERE entity.type IN ['PERSON', 'LOCATION', 'ORGANIZATION', 'EVENT']
CALL apoc.merge.node(
['Entity', apoc.text.capitalize(toLower(entity.type))],
{name: entity.name}, {}, {}
)
YIELD node AS e
MERGE (a)-[entityRel:GCP_ENTITY]->(e)
SET entityRel.score = entity.score
RETURN count(*)",
{ batchMode: "BATCH_SINGLE",
batchSize: 25,
params: {gcpApiKey: $gcpApiKey}}
)
YIELD batches, total, timeTaken, committedOperations, errorMessages, batch, operations
RETURN batches, total, timeTaken, committedOperations, errorMessages, batch, operations;
批次 | 总数 | 花费时间 | 已提交的操作 | 错误消息 | 批次 | 操作 |
---|---|---|---|---|---|---|
3 |
59 |
46 |
59 |
{} |
{total: 3, committed: 3, failed: 0, errors: {}} |
{total: 59, committed: 59, failed: 0, errors: {}} |
CALL apoc.periodic.iterate(
"MATCH (a:Article) WHERE not(exists(a.processed)) RETURN a",
"CALL apoc.nlp.azure.entities.stream([item in $_batch | item.a], {
key: $azureApiKey,
url: $azureApiUrl,
nodeProperty: 'body'
})
YIELD node AS a, value
SET a.processed = true
WITH a, value
UNWIND value.entities AS entity
WITH a, entity
WHERE entity.type IN ['Person', 'Organization', 'Location', 'Event']
CALL apoc.merge.node(
['Entity', apoc.text.capitalize(toLower(entity.type))],
{name: entity.name}, {}, {}
)
YIELD node AS e
MERGE (a)-[entityRel:AZURE_ENTITY]->(e)
SET entityRel.score = entity.score
RETURN count(*)",
{ batchMode: "BATCH_SINGLE",
batchSize: 25,
params: {azureApiUrl: $azureApiUrl, azureApiKey: $azureApiKey}}
)
YIELD batches, total, timeTaken, committedOperations, errorMessages, batch, operations
RETURN batches, total, timeTaken, committedOperations, errorMessages, batch, operations;
批次 | 总数 | 花费时间 | 已提交的操作 | 错误消息 | 批次 | 操作 |
---|---|---|---|---|---|---|
3 |
59 |
3 |
59 |
{} |
{total: 3, committed: 3, failed: 0, errors: {}} |
{total: 59, committed: 59, failed: 0, errors: {}} |
查询实体图
现在我们已经获得了实体,是时候查询实体图了。让我们首先返回每篇文章的实体,如下面的查询所示
MATCH (a:Article)
RETURN a.title AS title, [(a)-[:AWS_ENTITY]->(entity) | entity.name] AS entities
LIMIT 5;
标题 | 实体 |
---|---|
"曼联的卡瓦尼因 Instagram 上的'种族主义'帖子道歉" |
["Cavani", "FA", "Edinson Cavani", "Southampton", "Manchester United", "Football Association"] |
"脑震荡替补的必要性 - 足球周刊" |
["费伊·卡鲁瑟斯", "Soundcloud", "Mixcloud", "阿森纳", "巴里·格兰德宁", "Facebook", "伊万·默里", "劳尔·希门尼斯", "播客", "狼队", "大卫·路易斯", "Twitter", "Acast", "Stitcher", "南安普顿", "Audioboom", "曼联", "马克斯·拉什登", "拉尔斯·西弗特森"] |
"伦农成为凯尔特人未能延续国内统治地位的替罪羊" |
["流浪者", "莱斯特城", "布兰登·罗杰斯", "郡", "凯尔特人", "托尼·莫布雷", "罗斯", "戈登·斯特拉坎", "尼尔·伦农", "马丁·奥尼尔", "坦途", "联赛杯", "罗尼·德莱拉"] |
"那不勒斯通过复制马拉多纳的魔力战胜罗马,以此表达对他最真实的敬意 | 尼基·班迪尼" |
["圣保罗球场", "洛伦佐·因西涅", "欧洲", "那不勒斯", "罗马", "莱昂内尔·梅西", "那不勒斯", "迭戈·马拉多纳", "看台"] |
"脑震荡替补试验可能于明年初在英超联赛开始" |
["大卫·路易斯试验", "大卫·路易斯", "劳尔·希门尼斯", "墨西哥", "卫报", "阿森纳", "狼队", "英超联赛", "丹妮拉"] |
MATCH (a:Article)
RETURN a.title AS title, [(a)-[:GCP_ENTITY]->(entity) | entity.name] AS entities
LIMIT 5;
标题 | 实体 |
---|---|
"曼联的卡瓦尼因 Instagram 上的'种族主义'帖子道歉" |
["南安普顿", "问候", "俱乐部", "前锋", "事件", "曼联", "获胜", "身体", "乌拉圭人", "足总", "埃丁森·卡瓦尼", "朋友", "英格兰足球协会"] |
"脑震荡替补的必要性 - 足球周刊" |
["狼队", "Acast", "巴里·格兰德宁", "苹果", "伊万·默里", "胜利", "阿森纳", "Soundcloud", "费伊·卡鲁瑟斯", "马克斯·拉什登", "劳尔·希门尼斯", "Audioboom", "拉尔斯·西弗特森", "大卫·路易斯", "曼联"] |
"伦农成为凯尔特人未能延续国内统治地位的替罪羊" |
["布兰登·罗杰斯", "俱乐部", "赛季", "酒吧", "坦途", "流浪者", "莱斯特", "戈登·斯特拉坎", "罗尼·德莱拉", "比赛", "联赛杯", "失败", "城市经理", "球迷", "马丁·奥尼尔的", "凯尔特人", "托尼·莫布雷", "池塘", "尼尔·伦农", "罗斯郡"] |
"那不勒斯通过复制马拉多纳的魔力战胜罗马,以此表达对他最真实的敬意 | 尼基·班迪尼" |
["球迷", "洛伦佐·因西涅", "圣保罗球场", "那不勒斯", "那不勒斯", "家人", "迭戈·马拉多纳", "罗马", "获胜", "去世", "莱昂内尔·梅西", "欧洲", "球员"] |
"脑震荡替补试验可能于明年初在英超联赛开始" |
["俱乐部", "冲突", "丹妮拉", "劳尔·希门尼斯", "手术", "英超联赛", "大卫·路易斯", "球队", "阿森纳", "恢复", "卫报", "前锋", "墨西哥", "狼队", "规则变更", "大卫·路易斯试验"] |
MATCH (a:Article)
RETURN a.title AS title, [(a)-[:AZURE_ENTITY]->(entity) | entity.name] AS entities
LIMIT 5;
标题 | 实体 |
---|---|
"曼联的卡瓦尼因 Instagram 上的'种族主义'帖子道歉" |
["前锋", "埃丁森·卡瓦尼", "曼彻斯特联足球俱乐部", "乌拉圭国家足球队", "英格兰足球协会", "南安普顿足球俱乐部"] |
"脑震荡替补的必要性 - 足球周刊" |
["阿森纳足球俱乐部", "AudioBoom", "Stitcher Radio", "马克斯·拉什登", "拉尔斯·西弗特森", "曼彻斯特联足球俱乐部", "SoundCloud", "劳尔·希门尼斯", "Acast", "Facebook", "Mixcloud", "巴里·格兰德宁", "南安普顿评分", "伊万·默里", "Twitter", "南安普顿足球俱乐部", "大卫·路易斯", "Apple Podcasts", "费伊·卡鲁瑟斯"] |
"伦农成为凯尔特人未能延续国内统治地位的替罪羊" |
["马丁·奥尼尔", "莱斯特城足球俱乐部", "托尼·莫布雷", "罗尼·德莱拉", "流浪者足球俱乐部", "戈登·斯特拉坎", "联赛杯", "尼尔·伦农", "罗斯郡足球俱乐部", "英格兰足球联赛杯", "凯尔特人足球俱乐部", "布兰登·罗杰斯"] |
"那不勒斯通过复制马拉多纳的魔力战胜罗马,以此表达对他最真实的敬意 | 尼基·班迪尼" |
["那不勒斯", "洛伦佐·因西涅", "迭戈·马拉多纳", "罗马", "欧洲", "莱昂内尔·梅西", "莱昂内尔·梅西", "圣保罗球场", "那不勒斯足球俱乐部", "圣保罗球场"] |
"脑震荡替补试验可能于明年初在英超联赛开始" |
["英超联赛", "墨西哥", "丹妮拉", "卫报", "劳尔·希门尼斯", "狼队足球俱乐部", "大卫·路易斯", "阿森纳足球俱乐部", "卫报"] |
我们还可以使用文章对之间共有的实体来确定文章的相似性。如果我们想找到与加里·莱因克关于马拉多纳的视频类似的文章,我们可以编写以下查询
MATCH (a1:Article {title: "Gary Lineker: Maradona was 'like a messiah' in Argentina – video"})
MATCH (a1:Article)-[:AWS_ENTITY]-(entity)<-[:AWS_ENTITY]-(a2:Article)
RETURN a2.title AS otherArticle, collect(entity.name) AS entities
ORDER BY size(entities) DESC
LIMIT 5;
其他文章 | 实体 |
---|---|
"缅怀迭戈·马拉多纳:足球传奇人物去世,享年60岁——视频讣告" |
["世界杯", "迭戈·马拉多纳", "布宜诺斯艾利斯", "那不勒斯", "马拉多纳", "阿根廷", "巴塞罗那"] |
"经典 YouTube | 迭戈·阿曼多·马拉多纳,沉着冷静的射门和足球经理的孩子们" |
["英格兰", "世界杯", "迭戈·马拉多纳", "阿根廷", "马拉多纳", "莱因克", "加里·莱因克"] |
"阿根廷和那不勒斯的球迷哀悼迭戈·马拉多纳的去世——视频" |
["世界杯", "迭戈·马拉多纳", "布宜诺斯艾利斯", "那不勒斯", "阿根廷", "马拉多纳"] |
"天才的负担:马拉多纳让我们意识到年轻时达到巅峰会带来什么问题 | 维克·马克斯" |
["世界杯", "迭戈·马拉多纳", "墨西哥", "马拉多纳", "阿根廷"] |
"致敬迭戈·马拉多纳——足球周刊" |
["世界杯", "迭戈·马拉多纳", "布宜诺斯艾利斯", "墨西哥", "Twitter"] |
MATCH (a1:Article {title: "Gary Lineker: Maradona was 'like a messiah' in Argentina – video"})
MATCH (a1:Article)-[:GCP_ENTITY]-(entity)<-[:GCP_ENTITY]-(a2:Article)
RETURN a2.title AS otherArticle, collect(entity.name) AS entities
ORDER BY size(entities) DESC
LIMIT 5;
其他文章 | 实体 |
---|---|
"缅怀迭戈·马拉多纳:足球传奇人物去世,享年60岁——视频讣告" |
["巴塞罗那", "阿根廷", "那不勒斯", "布宜诺斯艾利斯"] |
"经典 YouTube | 迭戈·阿曼多·马拉多纳,沉着冷静的射门和足球经理的孩子们" |
["阿根廷", "英格兰", "加里·莱因克"] |
"迭戈·马拉多纳的私人医生否认对其去世负有责任" |
["家", "足球运动员", "布宜诺斯艾利斯"] |
"永葆青春的兹拉坦·伊布拉希莫维奇继续为米兰效力 | 尼基·班迪尼" |
["家", "那不勒斯"] |
"‘他带我们所有人去天堂’:足球迷对迭戈·马拉多纳去世的反应" |
["阿根廷", "布宜诺斯艾利斯"] |
MATCH (a1:Article {title: "Gary Lineker: Maradona was 'like a messiah' in Argentina – video"})
MATCH (a1:Article)-[:AZURE_ENTITY]-(entity)<-[:AZURE_ENTITY]-(a2:Article)
RETURN a2.title AS otherArticle, collect(entity.name) AS entities
ORDER BY size(entities) DESC
LIMIT 5;
其他文章 | 实体 |
---|---|
"缅怀迭戈·马拉多纳:足球传奇人物去世,享年60岁——视频讣告" |
["巴塞罗那足球俱乐部", "马拉多纳", "那不勒斯足球俱乐部", "那不勒斯", "阿根廷", "布宜诺斯艾利斯", "迭戈·马拉多纳"] |
"经典 YouTube | 迭戈·阿曼多·马拉多纳,沉着冷静的射门和足球经理的孩子们" |
["加里·莱因克", "阿根廷国家足球队", "谢菲尔德星期三足球俱乐部", "英格兰", "英格兰国家足球队", "阿根廷", "迭戈·马拉多纳"] |
"致敬迭戈·马拉多纳——足球周刊" |
["谢菲尔德星期三足球俱乐部", "马拉多纳", "墨西哥", "Twitter", "布宜诺斯艾利斯", "迭戈·马拉多纳"] |
"阿根廷和那不勒斯的球迷哀悼迭戈·马拉多纳的去世——视频" |
["马拉多纳", "那不勒斯足球俱乐部", "阿根廷", "布宜诺斯艾利斯", "迭戈·马拉多纳"] |
"正是马拉多纳的反抗精神最激励了我 | 肯南·马利克" |
["阿根廷国家足球队", "英格兰", "英格兰国家足球队", "迭戈·马拉多纳"] |
这些文章共有的某些实体实际上并没有意义(例如“家”或“Twitter”),但总的来说,这些文章将是加里·莱因克文章“推荐文章”部分的良好候选对象。
如果我们想进一步过滤提取的实体,我们可以尝试仅包含那些在维基百科中有条目的实体。有关该方法的更多详细信息,请参阅 教程:使用 NLP 和本体构建知识图谱。