Plume/plume-models/src
KITAITI Makoto 92a386277b
Switchable tokenizer (#776)
* [REFACTORING]Rename whitespace_tokenizer to tag_tokenizer for
registration

Name representing its purpose is preferred.

* Add lindera-tantivy to plume-model's dependencies

* Install lindera-tantivy

* Add SearchTokenizerConfig struct

* Add search tokenizers to config option

* Use CONFIG for tokenizers

* Use enum to hold tokenizer config instead of initializing on config phase

* Use guard instead of duplicate default values

* Use as_deref() instead of guard

* Move SearchTokenizer from plume-models to plume-models::search::tokenizer

* Rename SearchTokenizer to TokenizerKind

* Define SearchTokenierConfig::determine_tokenizer()

* Use determine_tokenizer in SearchTokenizerConfig::init()

* Pass tokenizer config to Searcher methods

* Add LowerCase filter to Lindera tokenizer

* Add test for Lindera tokenizer

* Define SEARCH_LANG env to specify tokenizers set

* Run cargo fmt

* Make Lindera tokenizer optional

* Fix typos
2020-06-17 16:57:28 +02:00
..
search Switchable tokenizer (#776) 2020-06-17 16:57:28 +02:00
timeline Don't show boosts and likes for "all" and "local" in timelines (#781) 2020-06-15 19:50:28 +02:00
admin.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
api_tokens.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
apps.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
blocklisted_emails.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
blog_authors.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
blogs.rs Switchable tokenizer (#776) 2020-06-17 16:57:28 +02:00
comment_seers.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
comments.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
config.rs Switchable tokenizer (#776) 2020-06-17 16:57:28 +02:00
db_conn.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
follows.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
headers.rs Add unit tests for main model parts (#310) 2018-11-24 12:44:17 +01:00
inbox.rs Add support for generic timeline (#525) 2019-10-07 19:08:20 +02:00
instance.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
lib.rs Switchable tokenizer (#776) 2020-06-17 16:57:28 +02:00
likes.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
lists.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
medias.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
mentions.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
migrations.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
notifications.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
password_reset_requests.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
plume_rocket.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
post_authors.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
posts.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
reshares.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
safe_string.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
schema.rs Email blocklisting (#718) 2020-01-12 19:41:35 +01:00
tags.rs Rust 2018! (#726) 2020-01-21 07:02:03 +01:00
users.rs Switchable tokenizer (#776) 2020-06-17 16:57:28 +02:00