Switchable tokenizer (#776)

* [REFACTORING]Rename whitespace_tokenizer to tag_tokenizer for
registration

Name representing its purpose is preferred.

* Add lindera-tantivy to plume-model's dependencies

* Install lindera-tantivy

* Add SearchTokenizerConfig struct

* Add search tokenizers to config option

* Use CONFIG for tokenizers

* Use enum to hold tokenizer config instead of initializing on config phase

* Use guard instead of duplicate default values

* Use as_deref() instead of guard

* Move SearchTokenizer from plume-models to plume-models::search::tokenizer

* Rename SearchTokenizer to TokenizerKind

* Define SearchTokenierConfig::determine_tokenizer()

* Use determine_tokenizer in SearchTokenizerConfig::init()

* Pass tokenizer config to Searcher methods

* Add LowerCase filter to Lindera tokenizer

* Add test for Lindera tokenizer

* Define SEARCH_LANG env to specify tokenizers set

* Run cargo fmt

* Make Lindera tokenizer optional

* Fix typos

This commit is contained in:

KITAITI Makoto

2020-06-17 16:57:28 +02:00

committed by

GitHub

parent 297d9fcf40

commit 92a386277b

13 changed files with 626 additions and 92 deletions

									
										plume-cli/Cargo.toml
									
		+1
		
												View File
												
				@@ -23,3 +23,4 @@ path = "../plume-models"

				[features]

				postgres = ["plume-models/postgres", "diesel/postgres"]

				sqlite = ["plume-models/sqlite", "diesel/sqlite"]

				search-lindera = ["plume-models/search-lindera"]