Vaex out-of-core DataFrames: analyze billions of rows without the RAM
Process massive tabular datasets with lazy, memory-mapped Vaex DataFrames — filtering, virtual columns, fast aggregations, big-data visualizations, format conversion, ML integration
- What
- Process massive tabular datasets with lazy, memory-mapped Vaex DataFrames — filtering, virtual columns, fast aggregations, big-data visualizations, format conversion, ML integration
- Cost
- Free
- Needs
- Python 3.10+ (3.12+ recommended); uv or pip to install vaex; a large tabular dataset to analyze — the skill is a guided reference, not software
- Install
- Copy the installer prompt below into your Muse — your agent does the rest.
Curated by Skill Harbor — @k-dense-ai's Vaex skill for analyzing tabular datasets too large for RAM — billions of rows processed at over a billion rows per second via lazy, memory-mapped, out-of-core DataFrames. Covers six capability areas: DataFrames and data loading (HDF5, CSV, Arrow, Parquet, pandas/NumPy conversion), data processing and manipulation (filtering, virtual columns, expressions, groupby aggregations), performance and optimization (lazy evaluation, delay=True batching, caching), data visualization (heatmaps, histograms, scatter plots via df.viz), machine learning integration (scalers, encoders, PCA, K-means, scikit-learn/XGBoost/CatBoost), and I/O (format recommendations, export strategies). Includes concrete common patterns: converting a large CSV to HDF5 for instant future loads, batching multiple aggregations with delay=True, and feature engineering with zero-memory-overhead virtual columns. States the Vaex-vs-alternatives rule honestly: polars when data fits in RAM and you need max in-memory speed, dask for distributed clusters, Vaex for single-machine out-of-core analytics. Honest caveats: requires Python 3.10+ (3.12+ recommended with vaex 4.19.0); the skill asks that substantial use be cited in manuscripts (arXiv:2609.00065). MIT licensed. Skill Harbor never reviews the code, review it yourself before use. Discovered via skills.sh.
Version:
Quick answer
Vaex out-of-core DataFrames is a free Muse skill for analyzing tabular datasets too large for RAM, processing billions of rows through lazy, memory-mapped DataFrames. It covers loading formats such as HDF5, CSV, Arrow and Parquet, plus filtering, virtual columns, expressions and groupby aggregations. It also adds visualizations, machine learning integration, format conversion and optimization with lazy evaluation, batching and caching.
Install
Copy the install package below, then paste it into Muse🧪 Install ran in our test sandbox · Nov 11, 2026📌 Pinned version: commit 92ace75Step 0 — before installing anything: read this install prompt, summarize in a few lines what this build does and what installing it involves, then ask me to confirm before you proceed. Prerequisites: Python 3.10+ (3.12+ recommended); uv or pip to install vaex; a large tabular dataset to analyze — the skill is a guided reference, not software Install "Vaex out-of-core DataFrames: analyze billions of rows without the RAM" for me. It gives my agent @k-dense-ai's Vaex playbook: load and convert large files (HDF5, CSV, Arrow, Parquet), filter and engineer features with zero-overhead virtual columns, batch fast aggregations, visualize billions of rows, build ML pipelines, and choose honestly between Vaex, polars and dask by workload. MIT-licensed. Repository: https://raw.githubusercontent.com/k-dense-ai/scientific-agent-skills/92ace75ac21efe19a620434e0ca4e356081fe807/skills/vaex/SKILL.md Pinned version: install the code at commit 92ace75ac21efe19a620434e0ca4e356081fe807 exactly (Skill Harbor tested this version). Do not substitute a newer version. If this commit is unreachable, stop and tell me. 1. Fetch the SKILL.md file (and any helper files) from the repository path into a temporary folder 2. Safety check: review the SKILL.md and scripts for anything suspicious (unexpected network calls, shell commands, credential harvesting). This repo should contain zero secrets in code, credentials only via the secure vault, allowed hosts declared in the SKILL.md. Verify that holds here; STOP on any red flag and tell me. 3. Install it as a skill: copy SKILL.md and its helper files into the agent's skills directory, in a folder named "vaex". 4. Verify with no network calls: frontmatter valid, files in place. 5. Report what was installed, where, and what I still need to do myself (e.g. install vaex via uv pip install vaex; point the agent at my large data files; cite the Scientific Agent Skills paper (arXiv:2609.00065) if it materially contributed to a manuscript). GitHub is optional: if I have a GitHub account or the gh CLI, you may use it; otherwise public access is fine. Never require it unless it's in the prerequisites above. Rules: don't touch anything outside the temp folder and the install target. If anything looks off, stop and ask me.
Saved to your recent installs. Find it anytime on /connect.
🔒 Security, provenance and what this touches
The install prompt below already includes the vetting steps: your agent follows the community checklist before installing anything with executable code. Want more?
Verification certificate
Dated facts about what was checked for this listing, and how. This is not a safety certification.
- Pinned version
- 92ace75 · 2026-10-13
- Sandbox install test
- Install ran in our test sandbox · Nov 11, 2026 · Observed during the install test: files.pythonhosted.org, pypi.org
- Creator identity
- Not verified
- Fingerprint watch
- Last checked · Nov 19, 2026 : no change detected in the files linked by this install prompt
- Install prompt analysis
- 2026-10-13 : No commands, network calls, packages, file writes or API keys detected.
Skill Harbor never reviews the code of a listed skill. These are dated facts about the install prompt and our install test, not a safety certification. Review a skill yourself before you install it. How we verify
Questions
How do I install a skill?
Every product page includes a copy-paste install prompt. Paste it into your Muse and it sets the skill up for you — no manual configuration.
Where does my money go?
Straight to the seller. Skill Harbor never processes payments: checkout happens on the seller’s own page, usually Stripe.
What does the ✓ next to a creator’s name mean?
It means we confirmed the identity of the person behind the listing. It says nothing about the code itself — always check a skill before installing it.