问题描述
将 Tika 解析器添加到 StormCrawler 时,不会从此字段中提取任何信息并存储在 ElasticSearch 中。
es-crawler.flux
includes:
- resource: true
file: "/crawler-default.yaml"
override: false
- resource: false
file: "crawler-conf.yaml"
override: true
- resource: false
file: "es-conf.yaml"
override: true
spouts:
- id: "spout"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.Aggregationspout"
parallelism: 10
- id: "filespout"
className: "com.digitalpebble.stormcrawler.spout.Filespout"
parallelism: 1
constructorArgs:
- "."
- "seeds.txt"
- true
bolts:
- id: "filter"
className: "com.digitalpebble.stormcrawler.bolt.URLFilterBolt"
parallelism: 1
- id: "partitioner"
className: "com.digitalpebble.stormcrawler.bolt.URLPartitionerBolt"
parallelism: 1
- id: "fetcher"
className: "com.digitalpebble.stormcrawler.bolt.FetcherBolt"
parallelism: 1
- id: "sitemap"
className: "com.digitalpebble.stormcrawler.bolt.SiteMapParserBolt"
parallelism: 1
- id: "parse"
className: "com.digitalpebble.stormcrawler.bolt.JSoupParserBolt"
parallelism: 1
- id: "tika_redirection"
className: "com.digitalpebble.stormcrawler.tika.RedirectionBolt"
parallelism: 1
- id: "tika_parser"
className: "com.digitalpebble.stormcrawler.tika.ParserBolt"
parallelism: 1
- id: "index"
className: "com.digitalpebble.stormcrawler.elasticsearch.bolt.IndexerBolt"
parallelism: 1
- id: "status"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt"
parallelism: 1
- id: "status_metrics"
className: "com.digitalpebble.stormcrawler.elasticsearch.metrics.StatusMetricsBolt"
parallelism: 1
streams:
- from: "spout"
to: "partitioner"
grouping:
type: SHUFFLE
- from: "spout"
to: "status_metrics"
grouping:
type: SHUFFLE
- from: "partitioner"
to: "fetcher"
grouping:
type: FIELDS
args: ["key"]
- from: "fetcher"
to: "sitemap"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "sitemap"
to: "parse"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "parse"
to: "index"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "fetcher"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "sitemap"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "parse"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "parse"
to: "tika_redirection"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "tika_redirection"
to: "tika_parser"
grouping:
type: LOCAL_OR_SHUFFLE
streamId: "tika"
- from: "tika_parser"
to: "index"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "tika_parser"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "index"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "filespout"
to: "filter"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "filter"
to: "status"
grouping:
streamId: "status"
type: CUSTOM
customClass:
className: "com.digitalpebble.stormcrawler.util.URLStreamGrouping"
constructorArgs:
- "byDomain"
我将这些设置添加到了 crawler-conf.yaml:
crawler-conf.yaml
parser.mimetype.whitelist:
- application/.*pdf.*
jsoup.treat.non.html.as.error: false
另外,我在运行拓扑时发现以下日志:
16:27:29.867 [Thread-43-tika_parser-executor[22,22]] INFO c.d.s.t.ParserBolt - skipped_trimmed -> http://cds.iisc.ac.in/wp-content/uploads/DS256.2017.Storm_.Tutorial.pdf
我更喜欢从 pdf 中提取所有可能的字段,并使用数组存储页面中的信息,因此,在 Elasticsearch 中,一页成为数组中的一个元素。
解决方法
请参阅 ParserBolt - 如果在获取期间修剪文档,则不会进行解析。
您可以在 conf 中禁用修剪
http.content.limit: -1
这应该会得到用 Tika 解析的文档。生成的元数据将带有前缀 parse.。您可能需要编写一个自定义 Bolt 以您想要的格式处理数据,即 ES 中每页一个键。