Switchable tokenizer (#776)
* [REFACTORING]Rename whitespace_tokenizer to tag_tokenizer for registration Name representing its purpose is preferred. * Add lindera-tantivy to plume-model's dependencies * Install lindera-tantivy * Add SearchTokenizerConfig struct * Add search tokenizers to config option * Use CONFIG for tokenizers * Use enum to hold tokenizer config instead of initializing on config phase * Use guard instead of duplicate default values * Use as_deref() instead of guard * Move SearchTokenizer from plume-models to plume-models::search::tokenizer * Rename SearchTokenizer to TokenizerKind * Define SearchTokenierConfig::determine_tokenizer() * Use determine_tokenizer in SearchTokenizerConfig::init() * Pass tokenizer config to Searcher methods * Add LowerCase filter to Lindera tokenizer * Add test for Lindera tokenizer * Define SEARCH_LANG env to specify tokenizers set * Run cargo fmt * Make Lindera tokenizer optional * Fix typos
This commit is contained in:
@@ -78,6 +78,7 @@ postgres = ["plume-models/postgres", "diesel/postgres"]
|
||||
sqlite = ["plume-models/sqlite", "diesel/sqlite"]
|
||||
debug-mailer = []
|
||||
test = []
|
||||
search-lindera = ["plume-models/search-lindera"]
|
||||
|
||||
[workspace]
|
||||
members = ["plume-api", "plume-cli", "plume-models", "plume-common", "plume-front", "plume-macro"]
|
||||
|
||||
Reference in New Issue
Block a user