v0.40.0-beta.2

Full Changelog: release-root/0.40.0-beta.N...v0.40.0-beta.2

v0.40.0-beta.1

What's Changed

Breaking Changes 🛠

• refactor: remove not used storage class and blob dataset by @Xuanwo in #5131

New Features 🎉

• feat: add fuzziness to json inverted match query by @wojiaodoubao in #5048
• feat(java): expose cleanup_with_policy api by @fangbo in #5136
• feat(java): supports building scalar indices distributedly in java module by @steFaiz in #4961

Bug Fixes 🐛

• fix: always return correct json schema to users by @Xuanwo in #5109
• fix: ensure I/O cancels correctly when scan is dropped by @westonpace in #5129
• fix: remove unnecessary object store initialization by @zhangyue19921010 in #5145
• fix: use typed LanceNamespace for python storage options provider by @jackye1995 in #5151

Performance Improvements 🚀

• perf: add a chunk cache to avoid decoding duplicated miniblock chunks by @niyue in #4846

Full Changelog: v0.39.0...v0.40.0-beta.1

v0.39.0

What's Changed

Breaking Changes 🛠

• feat!: incremental indexing via SPFresh by @BubbleCal in #4837

New Features 🎉

• feat(java): expose ManifestSummary to java api by @steFaiz in #5092
• feat: support dynamic storage options provider with AWS credentials vending by @jackye1995 in #4905

Bug Fixes 🐛

• fix: infinite kmeans if the largest cluster produces only 1 cluster by @BubbleCal in #5078
• fix: remove remainder explain_plan method in Python by @ddupg in #5085
• fix(rust): add explicit dependency on chrono serde feature by @wjones127 in #5110
• fix: no panic on unknown version by @wjones127 in #5111
• fix: skip compression in create_per_value if compression metadata is set to none by @wojiaodoubao in #5086
• fix: forward incompatibility of prerelease in writer version by @jackye1995 in #5116

Documentation 📚

• docs: minor doc fix for docs/src/format/file/encoding.md by @zhangyue19921010 in #5108
• docs: add fragment level update columns docs by @xloya in #5123

Other Changes

• refactor: introduce SchemaAdapter to perform logical/physical transform by @Xuanwo in #5096

Full Changelog: v0.38.3...v0.39.0

v0.38.3

What's Changed

Breaking Changes 🛠

• refactor!: cleanup public API, remove lance_arrow re-exports by @westonpace in #4991

New Features 🎉

• feat: handle forking better by @cmccabe in #4903
• feat: add multi-path support for lance data paths by @jaystarshot in #4765
• feat: support general compression zstd/lz4 in blocks by @lyang24 in #4900
• feat: fts supports custom stop words by @wojiaodoubao in #4866
• feat: implement add_bases api to add bases to lance dataset by @jaystarshot in #4945
• feat: allow a commit message to be specified in the python dataset commit method by @westonpace in #4952
• feat: support tracking newly inserted and updated rows between versions by @yanghua in #4741
• feat: support fetching _rowid and _rowaddr in take operation by @chunshao90 in #4794
• feat: support mTLS in REST namespace by @jackye1995 in #4981
• fix: populate all fields from ExecutionSummaryCounts to ScannerStats by @jaystarshot in #4980
• feat: support WhenMatched::Fail for MergeInsert by @ddupg in #4956
• feat(java): support shallow_clone by @majin1102 in #4962
• feat: add VariablePackedStruct defination by @Xuanwo in #4950
• feat: log version on dataset load event by @wjones127 in #4988
• feat(java): support blob api by @majin1102 in #4769
• feat(python): support shallow_clone by @majin1102 in #4949
• feat: expose storage options in dataset by @jackye1995 in #5016
• feat: should dictionary encode based on size by @yingjianwu98 in #4972
• feat: add variable packed struct support by @Xuanwo in #5003
• feat(python): support index for nested field by @jackye1995 in #5027
• feat(java): support branch operations by @majin1102 in #5012
• feat(python): support branch operations by @majin1102 in #5010
• feat: add json fts in python by @wojiaodoubao in #5020
• feat: add file audit events for deleting staging manifest paths in external commit handler by @wjones127 in #4989
• feat: implement create index and data replacement conflict rules by @westonpace in #5077
• feat: support fragment level update columns operation in Python by @xloya in #5001

Bug Fixes 🐛

• fix: let Java module use LanceFileVersion::Stable (#4558) by @ColdL in #4559
• fix: fts match query on column without inverted index by @wojiaodoubao in #4859
• fix: fix broken FTS example by replacing ROW_ID with DOC_ID by @niebayes in #4917
• fix: correctly record output_rows in filtered read with hard range_after by @westonpace in #4919
• fix: rewrap LanceFilterExec with_new_children by @wkalt in #4920
• fix: optimize_indices may unexpectly delete delta indices by @BubbleCal in #4931
• fix: rebuild HNSW graph while remapping it by @BubbleCal in #4941
• fix: be compatible to old pack metadata in 2.0 by @Xuanwo in #4964
• fix: filter with < current_date() should expand with correct time by @Xuanwo in #4963
• fix: use correct logic to detect old/new scheme in binary block decoder by @westonpace in #4966
• fix: support preview relase in writer version by @jackye1995 in #4974
• fix: index cache assumed_entry_size is inconsistent by @ddupg in #4975
• fix: clarify column names in missing column errors by @wkalt in #4942
• fix: the KMeans may result in all zeros centroids by @BubbleCal in #4977
• fix: full text search may miss some results by @BubbleCal in #4986
• fix: do not modify Lance schema when projecting system columns by @jackye1995 in #4997
• fix: handle empty batches in dictionary decode helper by @yingjianwu98 in #4995
• fix: coerce nested regexp_match to boolean in filters by @BubbleCal in #5019
• fix: update to respect file version from write params when writing fragments in java by @morales-t-netflix in #5014
• fix: general block decompression mismatch for Lance 2.2 dictionaries by @Xuanwo in #5025
• fix: handle List types in Substrait field counting by @LuQQiu in #5015
• fix: ensure limit cancels scan by @westonpace in #5032
• fix: don't panic in 2.1 if one page has nulls and the other doesn't by @westonpace in #5074
• fix: correct read column ordering in Fragment::update_columns by @wayneli-vt in #4983
• fix(java): the CompactionOptions is not serializable by @steFaiz in #4819
• fix: fix the deduplicated prefix in the opendal s3 store root setting by @xloya in #5082

Documentation 📚

• docs: add RabitQ in vector index spec by @BubbleCal in #4913
• docs: document the metadata cache by @westonpace in #4953
• docs: add docs for newly added options by @Xuanwo in #4976
• docs: minor fix for docs/src/format/table/index.md#Deletion by @zhangyue19921010 in #5033
• docs: minor doc fix for stable row id description by @zhangyue19921010 in #5047
• docs: add initial migration guide and document some steps for 0.39 by @westonpace in #5052
• docs: add migration guide for diff_meta and namespace by @jackye1995 in #5054
• docs: add json to full text search document by @wojiaodoubao in #4865

Other Changes

• refactor: move lance-namespace into lance repo by @jackye1995 in #4978
• refactor: use lance-io object store for dir namespace and improve builder by @jackye1995 in #5045
• refactor: don't add new with_... columns to scanner by @westonpace in #5007

New Contributors

• @yingjianwu98 made their first contribution in #4901
• @niebayes made their first contribution in #4917
• @zhangyue19921010 made their first contribution in #4915
• @dependabot[bot] made their first contribution in #4954
• @chunshao90 made their first contribution in #4794

Full Changelog: v0.38.2...v0.38.3

v0.38.3-beta.11

What's Changed

New Features 🎉

• feat(java): support branch operations by @majin1102 in #5012
• feat(python): support branch operations by @majin1102 in #5010
• feat: add json fts in python by @wojiaodoubao in #5020

Bug Fixes 🐛

• fix: ensure limit cancels scan by @westonpace in #5032
• fix: don't panic in 2.1 if one page has nulls and the other doesn't by @westonpace in #5074

Documentation 📚

• docs: minor doc fix for stable row id description by @zhangyue19921010 in #5047
• docs: add initial migration guide and document some steps for 0.39 by @westonpace in #5052
• docs: add migration guide for diff_meta and namespace by @jackye1995 in #5054
• docs: add json to full text search document by @wojiaodoubao in #4865

Other Changes

• refactor: use lance-io object store for dir namespace and improve builder by @jackye1995 in #5045
• refactor: don't add new with_... columns to scanner by @westonpace in #5007

Full Changelog: v0.38.3-beta.10...v0.38.3-beta.11

v0.38.3-beta.10

What's Changed

New Features 🎉

• feat: should dictionary encode based on size by @yingjianwu98 in #4972
• feat: add variable packed struct support by @Xuanwo in #5003
• feat(python): support index for nested field by @jackye1995 in #5027

Bug Fixes 🐛

• fix: update to respect file version from write params when writing fragments in java by @morales-t-netflix in #5014
• fix: general block decompression mismatch for Lance 2.2 dictionaries by @Xuanwo in #5025
• fix: handle List types in Substrait field counting by @LuQQiu in #5015

Documentation 📚

• docs: minor fix for docs/src/format/table/index.md#Deletion by @zhangyue19921010 in #5033

Full Changelog: v0.38.3-beta.9...v0.38.3-beta.10

v0.38.3-beta.9

What's Changed

Breaking Changes 🛠

• refactor!: cleanup public API, remove lance_arrow re-exports by @westonpace in #4991

New Features 🎉

• feat: expose storage options in dataset by @jackye1995 in #5016

Full Changelog: v0.38.3-beta.8...v0.38.3-beta.9

v0.38.3-beta.8

What's Changed

New Features 🎉

• feat: log version on dataset load event by @wjones127 in #4988
• feat(java): support blob api by @majin1102 in #4769
• feat(python): support shallow_clone by @majin1102 in #4949

Bug Fixes 🐛

• fix: coerce nested regexp_match to boolean in filters by @BubbleCal in #5019

Full Changelog: v0.38.3-beta.7...v0.38.3-beta.8

v0.38.3-beta.7

Full Changelog: v0.38.3-beta.6...v0.38.3-beta.7