When talking about Stream/Buffer, we have already started using require("fs") to introduce files The module does some operations

The file module is an encapsulation of underlying file operations, such as file reading and writing/opening and closing/delete adding, etc.

The biggest feature of the file module is that all methods are provided There are two versions of Synchronous and Asynchronous . Methods with sync suffix are all synchronous methods, and those without are all asynchronous methods

File common sense

File Permissions

Because you need to operate the file, you need to set the corresponding permissions. [Related tutorial recommendations: nodejs video tutorial, Programming teaching]

is mainly divided into three roles, file owner, The group to which the file belongs and other users

File permissions are divided into read, write, and execute. The numbers are expressed as 4/2/1 respectively. When there is no permission, it is expressed as 0

If execution is canceled Permissions mean that no files in the folder can be accessed, and you cannot cd to the folder

Use Linux commandsllYou can check the permissions of the files/folders in the directory

The first d represents the folder, - represents the file, followed by the file permissions // TODO: What does @ represent

File identification

In Node, the identification bit represents the operation mode of the file, readable/writable/readable and writable, etc., which can be arranged and combined

File descriptor

As mentioned in the previous content, the operating system will assign a numerical identifier called a file descriptor to each open file, and use these values ????to track specific document.

File descriptors generally start from 3, 0/1/2 represent standard input/standard output/error output respectively

Commonly used API

Some practices

Filtering the appropriate files in your project

const fs = require("fs");
const path = require("path");
const { promisify } = require("util");
const reg = new RegExp("(.ts[x]*|.js[x]*|.json)$");
const targetPath = path.resolve(__dirname, "../mini-proxy-mobx");

const readDir = (targetPath, callback) => {
    fs.readdir(targetPath, (err, files) => {
        if (err) callback(err);
        files.forEach(async (file) => {
            const filePath = path.resolve(__dirname, `${targetPath}/${file}`);
            const stats = await promisify(fs.stat)(filePath);
            if (stats.isDirectory()) {
                await readDir(filePath);
            } else {
                checkFile(filePath);
            }
        });
    });
};
const checkFile = (file) => {
    if (reg.test(file)) {
        console.log(file);
    }
};

readDir(targetPath, (err) => {
    throw err;
});

File copy

Problem: Need to copy the contents of file 1 to file 2

File API

You can use fs.readFile to read the file content, and then use fs.writeFile to write a new file

const fs = require("fs");
const path = require("path");

const sourceFile = path.resolve(__dirname, "../doc/Mobx原理及丐版實現(xiàn).md");
const targetFile = path.resolve(__dirname, "target.txt");

fs.readFile(sourceFile, (err, data) => {
    if (err) throw err;
    const dataStr = data.toString();
    fs.writeFile(targetFile, dataStr, (err) => {
        if (err) throw err;
        console.log("copy success~");
        process.exit(1);
    });
});

? Is there any problem with this? As we said in Stream, it requires a Click a little bit, otherwise the memory will be overwhelmed when the file is large.

Buffer Use

Use the fs.open method to open the file, obtain the file descriptor, and then call the fs.read/fs.write method to read and write to a specific location. Amount of data

const copyFile = (source, target, size, callback) => {
    const sourceFile = path.resolve(__dirname, source);
    const targetFile = path.resolve(__dirname, target);

    const buf = Buffer.alloc(size);
    let hasRead = 0; // 下次讀取文件的位置
    let hasWrite = 0; // 下次寫入文件的位置
    fs.open(sourceFile, "r", (err, sourceFd) => {
        if (err) callback(err);
        fs.open(targetFile, "w", (err, targetFd) => {
            if (err) throw callback(err);
            function next() {
                fs.read(sourceFd, buf, 0, size, hasRead, (err, bytesRead) => {
                    if (err) callback(err);
                    hasRead += bytesRead;
                    if (bytesRead) {
                        fs.write(targetFd, buf, 0, size, hasWrite, (err, bytesWrite) => {
                            if (err) callback(err);
                            hasWrite += bytesWrite;
                            next();
                        });
                        return;
                    }
                    fs.close(sourceFd, () => { console.log("關(guān)閉源文件"); });
                    fs.close(targetFd, () => { console.log("關(guān)閉目標文件"); });
                });
            }
            next();
        });
    });
};

Stream uses

const fs = require("fs");
const path = require("path");
const readStream = fs.createReadStream(
    path.resolve(__dirname, "../doc/Mobx原理及丐版實現(xiàn).md")
);
const writeStream = fs.createWriteStream(path.resolve("target.txt"));
readStream.pipe(writeStream);

File upload

Small file upload

// 上傳后資源的URL地址
const RESOURCE_URL = `http://localhost:${PORT}`;
// 存儲上傳文件的目錄
const UPLOAD_DIR = path.join(__dirname, "../public");

const storage = multer.diskStorage({
    destination: async function (req, file, cb) {
        // 設(shè)置文件的存儲目錄
        cb(null, UPLOAD_DIR);
    },
    filename: function (req, file, cb) {
        // 設(shè)置文件名
        cb(null, `${file.originalname}`);
    },
});

const multerUpload = multer({ storage });

router.post(
    "/uploadSingle",
    async (ctx, next) => {
        try {
            await next();
            ctx.body = {
                code: 1,
                msg: "文件上傳成功",
                url: `${RESOURCE_URL}/${ctx.file.originalname}`,
            };
        } catch (error) {
            console.log(error);
            ctx.body = {
                code: 0,
                msg: "文件上傳失敗",
            };
        }
    },
    multerUpload.single("file")
);

Large file upload

Main steps

1. 前端接收大文件，并進行切片處理
2. 將每份切片進行上傳處理
3. 后端接收到所有的切片，存儲所有切片到一個文件夾中
4. 將文件夾中的切片做合并，并對切片做刪除
5. 再次上傳統(tǒng)一文件時，能夠快速上傳

具體實現(xiàn)

1. 前端切片

   const BIG_FILE_SIZE = 25 * 1024 * 1024;
   const SLICE_FILE_SIZE = 5 * 1024 * 1024;
   
   const uploadFile = async () => {
       if (!fileList?.length) return alert("請選擇文件");
       const file = fileList[0];
       const shouldUpload = await verifyUpload(file.name);
       if (!shouldUpload) return message.success("文件已存在，上傳成功");
       if (file.size > BIG_FILE_SIZE) {
           // big handle
           getSliceList(file);
       }
       // // normal handle
       // upload("/uploadSingle", file);
   };
   const getSliceList = (file: RcFile) => {
       const sliceList: ISlice[] = [];
       let curSize = 0;
       let index = 0;
       while (curSize < file.size) {
           sliceList.push({
               id: shortid.generate(),
               slice: new File(
                   [file.slice(curSize, (curSize += SLICE_FILE_SIZE))],
                   `${file.name}-${index}`
               ),
               name: file.name,
               sliceName: `${file.name}-${index}`,
               progress: 0,
           });
           index++;
       }
       uploadSlice(sliceList);
       setSliceList(sliceList);
   };

   file 是一種特殊的 Blob 對象，可以使用 slice 進行大文件分割

   

2. 上傳切片

   const uploadSlice = async (sliceList: ISlice[]) => {
     const requestList = sliceList
         .map(({ slice, sliceName, name }: ISlice, index: number) => {
             const formData = new FormData();
             formData.append("slice", slice);
             formData.append("sliceName", sliceName);
             formData.append("name", name);
             return { formData, index, sliceName };
         })
         .map(({ formData }: { formData: FormData }, index: number) =>
             request.post("/uploadBig", formData, {
                 onUploadProgress: (progressEvent: AxiosProgressEvent) =>
                     sliceUploadProgress(progressEvent, index),
             })
         );
     await Promise.all(requestList);
   };

   根據(jù)切片構(gòu)建每個切片的 formData，將二進制數(shù)據(jù)放在 slice 參數(shù)中，分別發(fā)送請求。

   onUploadProgress 來處理每個切片的上傳進度

   // Client
   const storage = multer.diskStorage({
     destination: async function (req, file, cb) {
         const name = file?.originalname.split(".")?.[0];
         const SLICE_DIR = path.join(UPLOAD_DIR, `${name}-slice`);
         if (!fs.existsSync(SLICE_DIR)) {
             await fs.mkdirSync(SLICE_DIR);
         }
         // 設(shè)置文件的存儲目錄
         cb(null, SLICE_DIR);
     },
     filename: async function (req, file, cb) {
         // 設(shè)置文件名
         cb(null, `${file?.originalname}`);
     },
   });
   
   // Server
   router.post(
       "/uploadBig",
       async (ctx, next) => {
           try {
               await next();
               const slice = ctx.files.slice[0]; // 切片文件
               ctx.body = {
                   code: 1,
                   msg: "文件上傳成功",
                   url: `${RESOURCE_URL}/${slice.originalname}`,
               };
           } catch (error) {
               ctx.body = {
                   code: 0,
                   msg: "文件上傳失敗",
               };
           }
       },
       multerUpload.fields([{ name: "slice" }])
   );

3. 切片合并

   當我們所有的切片上傳成功之后，我們依舊希望是按著原始文件作為保存的，所以需要對切片進行合并操作

   // Client
   const uploadSlice = async (sliceList: ISlice[]) => {
   		// ...和上述 uploadSlice 一致
   		mergeSlice();
   };
   
   const mergeSlice = () => {
       request.post("/mergeSlice", {
           size: SLICE_FILE_SIZE,
           name: fileList[0].name,
       });
   };
   
   // Server
   router.post("/mergeSlice", async (ctx, next) => {
       try {
           await next();
           const { size, name } = ctx.request.body ?? {};
           const sliceName = name.split(".")?.[0];
           const filePath = path.join(UPLOAD_DIR, name);
           const slice_dir = path.join(UPLOAD_DIR, `${sliceName}-slice`);
           await mergeSlice(filePath, slice_dir, size);
           ctx.body = {
               code: 1,
               msg: "文件合并成功",
           };
       } catch (error) {
           ctx.body = {
               code: 0,
               msg: "文件合并失敗",
           };
       }
   });
   
   // 通過 stream 來讀寫數(shù)據(jù)，將 slice 中數(shù)據(jù)讀取到文件中
   const pipeStream = (path, writeStream) => {
       return new Promise((resolve) => {
           const readStream = fs.createReadStream(path);
           readStream.on("end", () => {
               fs.unlinkSync(path);   // 讀取完成之后，刪除切片文件
               resolve();
           });
           readStream.pipe(writeStream);
       });
   };
   
   const mergeSlice = async (filePath, sliceDir, size) => {
       if (!fs.existsSync(sliceDir)) {
           throw new Error("當前文件不存在");
       }
       const slices = await fs.readdirSync(sliceDir);
       slices.sort((a, b) => a.split("-")[1] - b.split("-")[1]);
       try {
           const slicesPipe = slices.map((sliceName, index) => {
               return pipeStream(
                   path.resolve(sliceDir, sliceName),
                   fs.createWriteStream(filePath, { start: index * size })
               );
           });
           await Promise.all(slicesPipe);
           await fs.rmdirSync(sliceDir);  // 讀取完成之后，刪除切片文件夾
       } catch (error) {
           console.log(error);
       }
   };

4. 上傳文件校驗

   當我們上傳一個文件的時候，先去判斷在服務(wù)器上是否存在該文件，如果存在則直接不做上傳操作，否則按上述邏輯進行上傳

   // Client
   const verifyUpload = async (name: string) => {
       const res = await request.post("/verify", { name });
       return res?.data?.data;
   };
   
   const uploadFile = async () => {
       if (!fileList?.length) return alert("請選擇文件");
       const file = fileList[0];
       const shouldUpload = await verifyUpload(file.name);
       if (!shouldUpload) return message.success("文件已存在，上傳成功");
       if (file.size > BIG_FILE_SIZE) {
           // big handle
           getSliceList(file);
       }
       // // normal handle
       // upload("/uploadSingle", file);
   };
   
   // Server
   router.post("/verify", async (ctx, next) => {
       try {
           await next();
           const { name } = ctx.request.body ?? {};
           const filePath = path.resolve(UPLOAD_DIR, name);
           if (fs.existsSync(filePath)) {
               ctx.body = {
                   code: 1,
                   data: false,
               };
           } else {
               ctx.body = {
                   code: 1,
                   data: true,
               };
           }
       } catch (error) {
           ctx.body = {
               code: 0,
               msg: "檢測失敗",
           };
       }
   });

   上述直接使用文件名來做判斷，過于絕對，對文件做了相關(guān)修改并不更改名字，就會出現(xiàn)問題。更應(yīng)該采用的方案是根據(jù)文件相關(guān)的元數(shù)據(jù)計算出它的 hash 值來做判斷。

   const calculateMD5 = (file: any) => new Promise((resolve, reject) => {
       const chunkSize = SLICE_FILE_SIZE
       const fileReader = new FileReader();
       const spark = new SparkMD5.ArrayBuffer();
       let cursor = 0;
       fileReader.onerror = () => {
           reject(new Error('Error reading file'));
       };
       fileReader.onload = (e: any) => {
           spark.append(e.target.result);
           cursor += e.target.result.byteLength;
           if (cursor < file.size) loadNext();
           else resolve(spark.end());
           
       };
       const loadNext = () => {
           const fileSlice = file.slice(cursor, cursor + chunkSize);
           fileReader.readAsArrayBuffer(fileSlice);
       }
       loadNext();
   });

   本文所有的代碼可以github上查看

   總結(jié)

   本文從文件常識/常用的文件 API 入手，重點講解了 Node 中 File 的相關(guān)實踐，最后使用相關(guān)內(nèi)容實現(xiàn)了大文件上傳。

   更多node相關(guān)知識，請訪問：nodejs 教程！

   The above is the detailed content of Let's talk in depth about the File module in Node. For more information, please follow other related articles on the PHP Chinese website!