pub struct Dictionary { /* private fields */ }Expand description
NNCP token dictionary mapping 16-bit symbol IDs (0..vocab_size) to raw word byte slices.
Implementations§
Source§impl Dictionary
impl Dictionary
Sourcepub fn from_words(words: Vec<Vec<u8>>) -> Self
pub fn from_words(words: Vec<Vec<u8>>) -> Self
Creates a dictionary directly from its ordered word byte entries.
Sourcepub fn load(path: impl AsRef<Path>) -> Result<Self, PreprocessError>
pub fn load(path: impl AsRef<Path>) -> Result<Self, PreprocessError>
Loads a .voc dictionary file in Bellard’s escaped line format.
§Errors
Returns PreprocessError on I/O failure, invalid escape sequences, or entries
longer than 4096 bytes.
Sourcepub fn from_reader<R: Read>(reader: R) -> Result<Self, PreprocessError>
pub fn from_reader<R: Read>(reader: R) -> Result<Self, PreprocessError>
Parses a .voc dictionary from any Read source.
§Errors
Returns PreprocessError on read errors, invalid escapes, or oversized entries.
Sourcepub fn save(&self, path: impl AsRef<Path>) -> Result<(), PreprocessError>
pub fn save(&self, path: impl AsRef<Path>) -> Result<(), PreprocessError>
Writes this dictionary to path in Bellard’s escaped line format.
§Errors
Returns PreprocessError::Io on write failure.
Sourcepub fn write_to(&self, writer: &mut impl Write) -> Result<(), PreprocessError>
pub fn write_to(&self, writer: &mut impl Write) -> Result<(), PreprocessError>
Serializes this dictionary to writer in Bellard’s escaped line format.
§Errors
Returns PreprocessError::Io on write failure.
Trait Implementations§
Source§impl Clone for Dictionary
impl Clone for Dictionary
Source§impl Debug for Dictionary
impl Debug for Dictionary
impl Eq for Dictionary
Source§impl PartialEq for Dictionary
impl PartialEq for Dictionary
impl StructuralPartialEq for Dictionary
Auto Trait Implementations§
impl Freeze for Dictionary
impl RefUnwindSafe for Dictionary
impl Send for Dictionary
impl Sync for Dictionary
impl Unpin for Dictionary
impl UnsafeUnpin for Dictionary
impl UnwindSafe for Dictionary
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more