rsync / rclone
Rsync Algorithm
Used for reducing the cost of a file transfer by avoiding the transfer of blocks that are already at the destination.
Use Case - Streaming file Sync
rsync is a utility for efficiently transferring and synchronizing files across computer systems, by checking the timestamp and size of files. It is commonly found on Unix-like systems and functions as both a file synchronization and file transfer program. The rsync algorithm is a type of delta encoding, and is used for minimizing network usage.Zlib may be used for additional compression, and SSH or stunnel can be used for data security
Rsync is typically used for synchronizing files and directories between two different systems. For example, if the commandrsync local-file user@remote-host:remote-fileis run, rsync will use SSH to connect asusertoremote-host.Once connected, it will invoke the remote host's rsync and then the two programs will determine what parts of the file need to be transferred over the connection.
Rsync can also operate in a daemon mode, serving and receiving files in the native rsync protocol (using the "rsync://" syntax).
Commands
alias cp='rsync -aP'
-P for a progress bar
rsync [OPTION]... SRC [SRC]... DEST
rsync -zvr source.jpg ~/Desktop/dest.jpg
-z for compression
-v for verbose
-r for recursive
rclone
Rclone*("rsync for cloud storage")*is a command line program to sync files and directories to and from different cloud storage providers.
-
azure blob storage
-
amazon s3
Commands
rclone config
rclone lsd remote:
rclone mkdir remote:container
rclone ls remote:container
rclone sync /home/local/directory remote:container
# copy from local to remote
rclone --config s3.conf copy --no-traverse remote:zenatix-postgres-backup/`date +%Y-%m-%d`/$dump_file.tar.gz .
# copy from remote to local
rclone copy --no-traverse . remote:ubuntu@52.220.109.7:/home/ubuntu/sources/zenalytix/media/images/deployment_photos
BackBlaze
b2 authorize-account
b2 list-buckets
b2 upload-file TimeMachineBackupDeepak Documents.zip Documents.zip