Using S3 to pipe data to disk

2 min read

Jun 12, 2023

QuickCode: Piping your Data from S3 to disk with Typescript

Recently I was building a project that required I download some large CSV files for processing (up to hundreds of MB in size) and I needed to save those files to disk.

When I first started trying to pipe the file I was getting the error: Property 'pipe' does not exist on type 'SdkStream<Readable | ReadableStream<any> | Blob | undefined>'. with @aws-sdk/types & @aws-sdk/client-s3 3.289.0

I wasnt able to find a quick fix online, so here was my solution that I ultimatly needed without adding in a ts-ignore:

import { Readable } from "stream";

const client = new S3Client({
  credentials: {
    accessKeyId: process.env.S3KEY,
    secretAccessKey: process.env.S3SECRET,
  },
  region: 'us-east-1',
});

const inputStreamMain = client.send(new GetObjectCommand({ Bucket: 'sample', Key: 'tmp1234.csv' }));
const inputStream = inputStreamMain.Body as Readable;

const output = new Promise((resolve, reject) => {
  const filePath = path.join(process.env.TMP_FOLDER, 'tmp.csv');
  const outputStream = fs.createWriteStream(filePath);

  // pipe the input stream to the disk
  inputStream.pipe(outputStream);

  outputStream.on("finish", () => {
    resolve(filePath);
  });
});

The key part being casting the Body as Readable from the stream package in node.

I hope this helps you with your error!