If you ever need to repeatedly upload to a remote server, here is the command line of the day:
rsync -zrptL --delete-after -e "ssh" --include=core --include=tags --exclude=.DS_Store --cvs-exclude /local/dir user@host:/remote/dir/
rsync sends only the files that have changed. I just tried rsync today for the first time and I’m impressed. Its far faster than all of the ftp based synchronization tools I’ve used. Unfortunately, it only goes in one direction. Here are the meanings of the options:
z – use compression
r – recurse directories
p – preserve permissions
t – preserve times
L – copy (flatten) symbolic links
–delete-after – gets rid of any files in the remote directory that are not in the local directory. It prevents old files from getting orphaned on the server.
Put any ssh parameters you need inside the quotes.
I develop on Mac OS X, so –exclude=.DS_Store gets rid of those annoying little files.
–cvs-exclude gets rid of all the crud that CVS leaves lying around.
–include=core and –include=tags are my very hard won lesson of today. My program has a core and a tags directory. Tags is a special directory for cvs and removed by the –cvs-exclude option. Any file or directory named core is also removed by this option. –include puts em back. The other –cvs-exclude patterns are less likely to collide. I think –include has to come before the –cvs-exclude on the command line.
The trailing slash (or absence of one) on the source and destination directories matter.
Feed
Subscribe to this blog…
rsync works both ways; pushing and pulling data.
rsync goes up and down. To sync from remote to local just reverse the source and destination.
Try this link for a nice shell script and a modified version of rsync that can handle HFS+ resource forks. .
Take a look at Unison:
http://www.cis.upenn.edu/~bcpierce/unison/
It’s for synchronizing and also implements the rsync-algo for efficient bandwidth usage.
Look into Unison. It’s like rsync, but works in both directions.
http://www.cis.upenn.edu/~bcpierce/unison/
[...] ter now, but I still decided that I didn’t want to try that route. Besides, I just got my automated deployment scripts working on the Mac. So Monday morni [...]
From the root of my project, I use:
rsync -avzu –exclude-from=rsync-exclude.txt * host:`pwd`
from my development server to the live server where the paths are the same.
I list a bunch of files in rsync-exclude.txt like images used only in production, files like CHANGELOG, etc. This file gets added to my CVS repository with each project.
I don’t use the –cvs-exclude because i tag my project before a release/upload and `cvs export -r TAGNAME project` so you won’t have CVS/ directories, etc.
The -n option is great and I encourage you use it before running it without. It will print you a list of all the files it wants to upload, allowing you to double check that you are about to upload anything by accident.
[...] Jeff Moore%u2019s Blog � Blog Archive � rsync to remote server via ssh Jeff Moore%u2019s Blog � Blog Archive � rsync to remote server vi [...]
–exclude-cvs considered harmful.
Well, it’s harmed me at least. Consider the case where you’re editing files on a ‘remote’ machine prior to committing them via a central machine ‘closer’ to the CVS repository(*).
Consider the following: you cvs update, rsync ‘away’ to another machine and do some editing there, cvs update again (whoops), then then rsync ‘back’ (disaster). You’ve now got ‘CVS/entries’ that say you’ve got the most recent files on the central system, but you’ve actually just rsync’d old versions of the files back. CVS will think you’re trying to ‘un-modify’ all the files back to where they were when you first CVS updated.
If you always keep the CVS meta files with the files they relate to, that sequence of events will simply result in CVS complaing you have state files.
Trust me. If you think you can simply just remember where you are and always update/sync in the right order, you’re kidding yourself. If you’re rsyncing checked-out CVS trees, its far too easy to mess up: blow the cost: sync your CVS meta data!
(*) pserver seems like a solution here, but consider the case where you need to test code on more than one platform *before* committing the code.
Thanks for this post, answers a bunch of questions I was having.
Thanks for this blog post. It helped me confirm the cause of a problem during our latest code release: the “core” folder was not rsync’d across to the live server because of the –cvs-exclude option. –include=core solved this
Hello all,
I need some help with rsync functionality for ‘files-from=’.
I tried to build automatic script, to getting file list (1), which are old more than one year .. and passing them in file to remote rsync(2). But without reason there is about 50% of unprocessed files, no error are returned. Its look like the whole file.list isn’t processed, but some files are processed.
(0)You need to have your public key in your authorized_hosts.(you can do ’ssh localhost’ whitout password)
cat /root/.ssh/id_dsa.pub >> /root/.ssh/authorized_keys
(1) this works (you can find result in file called files.list)
ssh -T localhost -o StrictHostKeyChecking=no “sshpass -p ‘remote_password’ ssh remote_server -p 22 -o ConnectTimeout=2 -o StrictHostKeyChecking=no -o PasswordAuthentication=yes -o NumberOfPasswordPrompts=1 ‘find /var/spool/asterisk/monitor/ -type f -mtime +365 -exec echo \{\} \; ‘” 2>&1> files.list
(2) this doesn’t work (you can see in rsync.log, that there are no all files processed but why ?
rsync –bwlimit=2000 –timeout=2 –files-from=files.list -avH –rsh=”sshpass -p ‘remote_password’ ssh -l root” remote_server:/ /mnt/backup/ 2>&1> rsync.log
echo $? // returning (0)
PS: when you try ‘retarded’ command that don’t use ssh remote, whole files.list is procesed / moved
rsync –files-from=files.list -avH / /mnt/backup/ 2>&1> rsync.log
Any ideas how to solve this (strange issue)?
After 2 hours, I will be at home. There I will be able to say more on this.
Hey! I know this is kinda off topic however I’d figured I’d ask. Would you be interested in exchanging links or maybe guest authoring a blog post or vice-versa? My site discusses a lot of the same subjects as yours and I think we could greatly benefit from each other. If you’re interested feel free to shoot me an e-mail. I look forward to hearing from you! Great blog by the way!
I’ve said that least 4268918 times. The problem this like that is they are just too compilcated for the average bird, if you know what I mean
hi, demonstration gratitude you your script appearance is impressive. exactly saved your set on bing. skyway blow later for affirmatory
hey, i found your blog upon yahoo.it’s certainly hale written also it helped me lots. continue the suitable work.
Thanks for the good article. I even have learned from this article, Love the site!
To purchase online. This item has been discounted at this link.| Black Friday Sale Philips AS351/37 Fidelio Docking Speaker for Android | Black Friday Philips AS351/37
I follow
I am usually to blogging and that i truly recognize your content. The article has really peaks my interest. I’m progressing to bookmark your site and hold checking for whole fresh data.
Thank you for your nice post! it’s been terribly helpful. I hope that you simply will continue sharing your information with us.
Good article and right to the purpose. I don’t know if this is often truly the most effective place to ask however does one individuals have any thoughts on where to hire some skilled writers? Thanks before
good job thanks for the post
Ha, fun to learn something new. Tho I’m not sure I agree with the point. Nonetheless, share more
good work thanks for the publish
an attention-grabbing viewpoint!
a useful and helpful however of info
Heya i am for the first time here. I came across this board and I find It truly helpful & it helped me out much. I am hoping to give something back and help others such as you aided me.
Interesting.
I guess they analyzed that they might get the half percent passed, otherwise why not ask for 1 percent?
Cool info it is without doubt. My father has been searching for this update.
As I hyperlink somebody i consider the volume concern here is rattling blow up, value anybody for your causes. You should relate it up perpetually! fortunate of fortune.