Writing to SD cards looks straightforward until you hit power interruptions. I lost enough logs to treat write integrity as a first-class feature.

My format is append-only CSV with periodic file sync. I avoid rewriting headers or metadata during runtime. Each row includes timestamp, sensor IDs, and a checksum column that lets me detect partial lines after reboot.

I also rotate files by size and day. Smaller files recover faster, and data processing scripts run more predictably when logs are chunked.

The final piece is startup repair. On boot, the logger scans the tail of the last file, truncates invalid trailing bytes, and resumes with a new row marker. That keeps datasets usable even after abrupt shutdowns.

Append-only logging with periodic sync

The core loop keeps the file open, appends rows, and only forces a physical flush every few seconds. Flushing on every row wears the card and kills throughput; never flushing loses everything on power cut. A periodic flush() (which on the SD/SdFat libraries syncs the file's directory entry and data) is the compromise I settled on.

#include <SD.h>

const uint8_t   SD_CS         = 10;
const char     *LOG_PATH      = "/log.csv";
const uint32_t  FLUSH_EVERY_MS = 5000;

File logFile;
uint32_t lastFlush = 0;

// timestamp,sensorId,value with a simple XOR checksum column.
void logRow(uint32_t ts, uint8_t sensorId, float value) {
  char line[48];
  int n = snprintf(line, sizeof(line), "%lu,%u,%.2f",
                   (unsigned long)ts, sensorId, value);

  uint8_t chk = 0;
  for (int i = 0; i < n; i++) chk ^= (uint8_t)line[i];

  logFile.print(line);
  logFile.print(',');
  logFile.println(chk, HEX);

  if (millis() - lastFlush >= FLUSH_EVERY_MS) {
    logFile.flush();          // push buffered data + directory entry to the card
    lastFlush = millis();
  }
}

Boot-time integrity check

Before appending, I confirm the card mounts and the log opens for append. If the last line looks truncated (no newline, or a bad checksum), I write a fresh row marker so downstream parsers can skip the damaged tail instead of choking on it.

bool startLogger() {
  if (!SD.begin(SD_CS)) {
    Serial.println(F("SD init failed"));
    return false;
  }

  logFile = SD.open(LOG_PATH, FILE_WRITE);   // FILE_WRITE seeks to end (append)
  if (!logFile) {
    Serial.println(F("Cannot open log for append"));
    return false;
  }

  // If the previous run died mid-line, mark a clean boundary.
  if (logFile.size() > 0) {
    logFile.println(F("# reboot"));
    logFile.flush();
  }

  lastFlush = millis();
  return true;
}

The # reboot marker is cheap insurance: a partial trailing line from the last power cut is left in place, and every parser I use already ignores lines starting with #, so the dataset stays machine-readable without a separate repair pass.

Sources

  • Imported from the previous version of the site (gaborl.hu). Original publication around 2025-01-26.