Crate narrow

An experimental (work-in-progress) statically typed implementation of Apache Arrow.

This crate provides methods to automatically generate types to support reading and writing instances of abstract data types in Arrow’s in-memory data structures.

Why

• The arrow crate provides APIs that make sense when the array types are only known at run-time. Many of its APIs require the use of trait objects and downcasting. However, for applications where types are known at compile-time, these APIs are not ergonomic.
• Builders for nested array types are complex and error-prone.

There are other crates that aim to prevent users from having to maintain array builder code by providing derive macros. These builders typically produce type-erased arrays, whereas this crate only provides fully statically typed arrays.

Goals and non-goals

Goals

• Provide production ready, fully statically typed, safe and efficient Arrow array implementations
• Enable everyone using Rust to easily benefit from the Arrow ecosystem
• Provide zero-copy interop with the arrow crate
• Support custom buffer implementations e.g. to support accelerators
• Explore expressing Arrow concepts using the Rust type system, and mapping Rust concepts to Arrow

Non-goals

• Support arbitrary array types at runtime (the arrow crate supports this use case)
• Provide compute kernels
• Replace other Arrow implementations

Example

use narrow::{
    array::{StructArray, UnionArray},
    ArrayType, Length,
};

#[derive(ArrayType, Default, Clone, Debug, PartialEq, Eq)]
struct Foo {
    a: bool,
    b: u32,
    c: Option<String>,
}

#[derive(ArrayType, Default, Clone, Debug, PartialEq, Eq)]
struct Bar(Vec<u8>);

#[derive(ArrayType, Clone, Debug, PartialEq, Eq)]
enum FooBar {
    Foo(Foo),
    Bar(Bar),
    None,
}

let foos = vec![
    Foo {
        a: false,
        b: 0,
        c: None,
    },
    Foo {
        a: true,
        b: 42,
        c: Some("hello world".to_owned()),
    },
];
let struct_array = foos.clone().into_iter().collect::<StructArray<Foo>>();
assert_eq!(struct_array.len(), 2);
assert!(struct_array.0.a.iter().any(|x| x));
assert_eq!(struct_array.0.b.iter().sum::<u32>(), 42);
assert_eq!(struct_array.0.c.iter().filter_map(|x| x).collect::<String>(), "hello world");
assert_eq!(struct_array.into_iter().collect::<Vec<_>>(), foos);

let foo_bars = vec![
    FooBar::Foo(Foo {
        a: true,
        b: 42,
        c: Some("hello world".to_owned()),
    }),
    FooBar::Bar(Bar(vec![1, 2, 3, 4])),
    FooBar::None,
    FooBar::None,
];
let union_array = foo_bars
    .clone()
    .into_iter()
    .collect::<UnionArray<FooBar, 3>>();
assert_eq!(union_array.len(), 4);
assert_eq!(union_array.into_iter().collect::<Vec<_>>(), foo_bars);

Features

The crate supports the following optional features:

Derive support

• derive: adds ArrayType derive support.

Interop

• arrow-rs: array conversion methods for arrow.

Additional ArrayType implementations

• chrono: support for some chrono types.
• map: support for std::collections::HashMap.
• uuid: support for uuid::Uuid.

Docs

• Docs (release)
• Docs (main)

Minimum supported Rust version

The minimum supported Rust version for this crate is Rust 1.85.1.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Re-exports

pub use arrow_array;arrow-rs

pub use arrow_buffer;arrow-rs

pub use arrow_schema;arrow-rs

Modules

array

Sequences of values with known length all having the same type.

arrowarrow-rs

Interop with the arrow-rs crate.

bitmap

A collection of bits.

buffer

Traits for memory buffers.

logical

Logical array support.

offset

Offsets for variable-sized arrays.

Structs

NonNullable

Non-nullable types.

Nullable

Nullable types.

Traits

FixedSize

Subtrait for fixed-size types.

Index

Index operation for shared access to values in a collection.

Length

The length (or number of elements) of a collection.

Nullability

Nullability trait for nullable and non-nullable type constructors

Derive Macros

ArrayTypederive

Derive macro for the ArrayType trait.