To run data transfer tasks from the command line, the key is to use the right tools and master the basic command format. 1. Use rsync to perform local or remote synchronization, support breakpoint continuous transmission, compression and retain meta information. Common commands such as rsync -avz /source/ /dest/; 2. AWS users can upload and download files with aws s3 cp, and support recursive replication and parameter optimization after configuring credentials; 3. Google Cloud users can use gsutil cp to upload files or directories, and support concurrent settings to improve speed; 4. Other platforms can use azcopy (Azure), lftp (FTP/SFTP) or rclone (cross-platform); 5. It is recommended to combine scripts and cron automation tasks to enable compression or shard processing of large files, select tools that support breakpoint continuous transmission to deal with unstable networks, and prioritize log checking when errors occur.
To run data transfer tasks from the command line, the key is to use the right tools and master the basic command format. Most modern systems provide command line interfaces to perform such operations, such as rsync
, scp
, azcopy
(for Azure), gsutil
(Google Cloud), or aws s3 cp
(AWS). Below are some common scenarios and practical suggestions.
Use rsync for local or remote synchronization
If you need to copy between local directories, or synchronize files with remote servers, rsync
is a powerful and common choice.
- Support breakpoint continuous transmission
- Can compress data transmission
- Able to retain meta information such as permissions, timestamps, etc.
Common command examples:
rsync -avz /source/directory/ user@remote:/destination/directory/
illustrate:
-
-a
: Archive mode, retain all attributes -
-v
: Show detailed process -
-z
: Enable compression
If it's just local copy, remove the remote address:
rsync -avz /source/directory/ /backup/directory/
Tips: Before running for the first time, you can add
--dry-run
to see which files will be operated to avoid misoperation.
Upload or download files using the aws s3 command
If you are using AWS S3, you can directly use the command line after installing the AWS CLI.
Make sure that the credentials are configured:
aws configure
Then you can upload the file:
aws s3 cp myfile.txt s3://mybucket/
Or upload the entire directory recursively:
aws s3 cp myfolder/ s3://mybucket/myfolder/ --recursive
You can also add some parameters to optimize transmission, such as:
-
--storage-class STANDARD_IA
Specify storage type -
--sse
Enable encryption -
--quiet
silent execution does not output logs
Basic usage of gsutil for Google Cloud
For Google Cloud users, gsutil
is a standard data transfer tool.
Upload individual files:
gsutil cp myfile.txt gs://mybucket/
Upload directory:
gsutil cp -r myfolder gs://mybucket/
You can also set the concurrency option to increase the speed:
gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp bigfile gs://mybucket/
Note: If you are running on a GCP virtual machine, it is best to use a service account authorization instead of a personal account, which is safer and more convenient to automate.
Tips for other platforms or tools
Different platforms have different command line tools, but the logic of using is similar:
- Azure : Use
azcopy
to support multiple cloud platforms and have good performance. - FTP/SFTP : You can use
lftp
or scriptedsftp
commands. - Cross-platform synchronization : consider
rclone
and supports dozens of cloud services.
Several general suggestions:
- Daily maintenance tasks can be automatically executed using shell script cron
- Large file transfer recommended to enable compression or sharding
- If the network is unstable, select a tool that supports breakpoint continuous transmission
- Remember to check the log when an error occurs, and don't try again blindly
Basically that's it. As long as you choose the right tool and match it with the appropriate parameters, the command line data transmission is actually quite simple, but some details are easy to ignore.
The above is the detailed content of How to run a Data Transfer job from the command line?. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undress AI Tool
Undress images for free

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

ForNavicat,opentheseports:1)MySQL:3306,2)PostgreSQL:5432,3)Oracle:1521,4)SQLServer:1433,5)MongoDB:27017;useaPythonscripttocheckiftheyareopen,andensurefirewallsettingsallowtrafficontheseportsforsmoothdatabaseconnectivity.

The most direct way to view database properties is to use database management tools or execute specific commands. For MySQL, you can use SHOWDATABASES and SHOWCREATEDATABASE commands; PostgreSQL supports \l meta commands and SELECT to query the pg_database table; SQLServer can query the sys.databases system view. Graphical tools such as MySQLWorkbench, pgAdmin and SSMS also provide intuitive interfaces to view properties. Notes include permission control, version differences and restrictions in cloud service environments. After mastering these methods, you can easily obtain data regardless of whether you use the command line or the graphical interface.

To copy the table structure without copying data, use SQL commands or graphics tools. ① Use CREATETABLEnew_tableLIKEoriginal_table in MySQL; copy structure and index; ② You can also use CREATETABLEnew_tableASSELECT*FROMoriginal_tableWHERE1=0; but the primary key and index may be lost; ③ PostgreSQL supports CREATETABLEnew_table(LIKEoriginal_tableINCLUDINGALL); ④ SQLServer can use SELECTINTO to combine WHERE1

NavicatPremiumisthemostfeature-richedition,supportingmultipledatabasesandofferingallavailabletools.1.ItsupportsMySQL,MariaDB,PostgreSQL,SQLite,Oracle,MongoDB,andSQLServer,idealforusersworkingacrossvariousdatabases.2.Itincludesadvancedfeatureslikevisu

How to create a Sequence in Navicat? If you use a database that supports Sequence such as PostgreSQL or Oracle, you can use the following steps: 1. Open Navicat and connect to the database; 2. Find "Sequences" in the object tree on the left and right-click to select "New Sequence"; 3. Fill in the sequence name, starting value, minimum value, maximum value, step size, and whether to loop; 4. After saving, you can view the generated statement in the SQL panel. Sequence is different from the self-increment field. It is an independent object that can be shared across tables and is suitable for multi-table shared numbering scenarios. Sequence can be called by nextval() function when inserting data, or field defaults can be set when creating tables.

To create a new database connection in Navicat, it is actually not difficult. The key is to fill in a few key information. As long as you have the database address, port, account number and password, you can basically do it. The following are a few steps to explain how to operate, which is suitable for users who use Navicat for the first time. Basic steps to create a new connection After opening the Navicat main interface, click the "New connection" button. Next, a window will pop up to let you choose the database type, such as MySQL, PostgreSQL, SQLServer, etc. After selecting the right type, start filling in the connection information. The main contents that need to be filled in include: Connection name: Give yourself an easy-to-recognize name, such as "local test library" or "production"

Setting up timing tasks in Navicat must be implemented through the database event scheduler. The specific steps are as follows: 1. Confirm that the database has enabled the event scheduling function, use SHOWVARIABLESLIKE'event_scheduler' to check the status, if OFF, execute SETGLOBALevent_scheduler=ON to enable; 2. Create an event in Navicat, right-click the "Event" node and select "New Event", set the name, execution time and cycle, enter the SQL statement to be executed on the "Definition" page and save it; 3. Check the event status and next execution time, and can manually test by right-clicking "Run Events", check the log or mysql.even if an error occurs.

To add users, you need to invite others to register and set permissions through the sharing function. The permissions are divided into read-only and editable. If you remove users, delete the corresponding members through the sharing settings. Specific steps: 1. When adding a user, right-click to connect and select "Share" and enter the other party's email address; 2. Select read-only or editable mode when setting permissions; 3. Remove the user and enter the sharing option and click "Remove". It is recommended to use the company's email to register uniformly, check the shared content regularly, and cancel temporary collaboration permissions in a timely manner to ensure security.
