name: data-load description: Load data from files (CSV, JSON, JSONL, Parquet) or stdin for analysis and visualization with ggterm. Use when reading datasets, importing data, opening files, or when the user mentions loading, reading, or opening data. allowed-tools: Bash(bun:), Bash(npx:), Read, Write

Data Loading for ggterm

Load data into arrays of records for use with ggterm plotting and analysis.

Quick Patterns by Format

CSV

import { parse } from 'csv-parse/sync'
import { readFileSync } from 'fs'

const text = readFileSync('data.csv', 'utf-8')
const data = parse(text, {
  columns: true,        // First row as headers
  cast: true,           // Auto-convert numbers
  skip_empty_lines: true
})

Alternative with d3-dsv (lighter weight):

import { csvParse, autoType } from 'd3-dsv'

const data = csvParse(readFileSync('data.csv', 'utf-8'), autoType)

JSON

import { readFileSync } from 'fs'

// JSON array
const data = JSON.parse(readFileSync('data.json', 'utf-8'))

JSONL (Newline-delimited JSON)

const data = readFileSync('data.jsonl', 'utf-8')
  .trim()
  .split('\n')
  .map(line => JSON.parse(line))

From stdin (Piped Data)

// Bun
const input = await Bun.stdin.text()
const data = JSON.parse(input)

// Node.js
import { stdin } from 'process'
let input = ''
for await (const chunk of stdin) input += chunk
const data = JSON.parse(input)

From URL

const response = await fetch('https://example.com/data.json')
const data = await response.json()

TSV (Tab-separated)

import { tsvParse, autoType } from 'd3-dsv'

const data = tsvParse(readFileSync('data.tsv', 'utf-8'), autoType)

Type Coercion

ggterm expects numeric values for position aesthetics. Ensure proper typing:

const typed = data.map(row => ({
  ...row,
  // Convert date strings to timestamps
  date: new Date(row.date).getTime(),
  // Ensure numeric values
  value: Number(row.value),
  // Handle missing values
  score: row.score != null ? Number(row.score) : null
}))

Common Type Issues

Verification

After loading, always verify the data structure:

console.log(`Loaded ${data.length} rows`)
console.log('Columns:', Object.keys(data[0]))
console.log('Sample row:', data[0])

// Check for type issues
const numericCols = ['value', 'count', 'score']
for (const col of numericCols) {
  const nonNumeric = data.filter(r => typeof r[col] !== 'number')
  if (nonNumeric.length > 0) {
    console.warn(`${col}: ${nonNumeric.length} non-numeric values`)
  }
}

Installing Dependencies

If needed, install data loading libraries:

# For CSV parsing
bun add csv-parse
# or
bun add d3-dsv

# For Parquet (if needed)
bun add parquet-wasm

Integration with ggterm

Once data is loaded, pass directly to ggterm:

import { gg, geom_point } from '@ggterm/core'

const data = loadData('measurements.csv')

const plot = gg(data)
  .aes({ x: 'time', y: 'value' })
  .geom(geom_point())

console.log(plot.render({ width: 80, height: 24 }))

Large Files

For large files, consider streaming or sampling:

// Sample every Nth row
const sampled = data.filter((_, i) => i % 10 === 0)

// Or take first N rows for exploration
const preview = data.slice(0, 1000)