pub struct KvCache { /* private fields */ }Expand description
Layer-indexed key/value cache for autoregressive generation.
Implementations§
Source§impl KvCache
impl KvCache
Sourcepub fn new(num_layers: usize) -> Self
pub fn new(num_layers: usize) -> Self
Allocate an empty cache for num_layers transformer blocks.
Sourcepub fn past_len(&self, layer_idx: usize) -> usize
pub fn past_len(&self, layer_idx: usize) -> usize
Number of tokens currently retained in the cache window at layer_idx.
Sourcepub fn seq_pos(&self, layer_idx: usize) -> usize
pub fn seq_pos(&self, layer_idx: usize) -> usize
Total number of sequence positions processed so far at layer_idx (unaffected by sliding-window eviction).
Sourcepub fn append(
&mut self,
layer_idx: usize,
k: &Tensor,
v: &Tensor,
sliding_window: Option<usize>,
) -> Result<(Tensor, Tensor)>
pub fn append( &mut self, layer_idx: usize, k: &Tensor, v: &Tensor, sliding_window: Option<usize>, ) -> Result<(Tensor, Tensor)>
Append (k, v) of shape (batch, kv_heads, seq_len, head_dim) and return the concatenated history.
§Errors
Returns an error if layer_idx is out of bounds or concatenation fails.
Trait Implementations§
Auto Trait Implementations§
impl !RefUnwindSafe for KvCache
impl !UnwindSafe for KvCache
impl Freeze for KvCache
impl Send for KvCache
impl Sync for KvCache
impl Unpin for KvCache
impl UnsafeUnpin for KvCache
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> ErasedDestructor for Twhere
T: 'static,
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
impl<T, U> Imply<T> for U
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more