Using Robocopy for backups

Anything and everything software related that doesn't fit above can go in here!
Post Reply
User avatar
bubba
Staff Writer
Staff Writer
Posts: 4765
Joined: Sun May 01, 2005 10:24 am
Location: STL

Using Robocopy for backups

Post by bubba »

Ok, not sure if anyone has used this before. I have mentioned it in some threads, but just got done explaining it in an email thought I would put it here.

Robocopy is a command that is available in Vista and Win7 by default, and XP by installing the Server 2003 resource kit. It works a lot like XCOPY from back in the day, but has a bit more control. I use it for my backups to pull from one place, copy to another for centralized offloading, no funky compression, or file formats, like with most backup software.

What I like about Robocopy is that its smart. It will look to see if a file has changed/deleted in the source and make the change in the destination. You can also set it to ignore files.

For my backups I have the following

Code: Select all

ROBOCOPY \\192.168.2.160\cad c:\Backup\cad /s /MIR /xf *.bak /xd \\192.168.2.160\cad\trashbox /s /R:0 /W:0 /PURGE  /LOG:cad.txt /TEE /NFL /NDL
This copy's everything from CAD folder on my NAS to my local machine for offloading to DVD for archiving. It ignores the .BAK files (files that AutoCAD generates with every file save) and the folder "trashbox", then purges everything that doesn't match the source folder. Then spits out a report in a text file that has info showing what it did and how long it took.

here is what the report looks like, the "extra files" are the one that are no longer in the source since the last time the back up was done. It took the Robocopy all of 18 seconds to sift through over 49,000 files in 1,000 folders. Now how long this takes will change depending on the file sizes of stuff to be copied/deleted, but its a little faster then recopying the whole 4.5GB.

Code: Select all

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows                              
-------------------------------------------------------------------------------

  Started : Thu Feb 04 06:42:25 2010

   Source : \\192.168.2.160\cad\
     Dest : c:\Backup\cad\

    Files : *.*
	    
Exc Files : *.bak
	    
 Exc Dirs : \\192.168.2.160\cad\trashbox
	    
  Options : *.* /NDL /NFL /TEE /S /COPY:DAT /PURGE /MIR /R:0 /W:0 

------------------------------------------------------------------------------

	  *EXTRA File 		       0	c:\Backup\cad\New Text Document.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      1029         0      1029         0       0        0
   Files :     49590         0     49590         0       0        1
   Bytes :   4.469 g         0   4.469 g         0       0        0
   Times :   0:00:18   0:00:00                       0:00:00   0:00:18

   Ended : Thu Feb 04 06:42:43 2010
I find this is very handy for collections of large amounts of little files like images, music, documents, and so on. Make a batch file and set up a timer in the scheduler pointed to the batch file and it will do it at the time you set.

Here is list of all the switches for the command, now you can see the flexibility of the command.

Code: Select all

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Thu Feb 04 07:21:04 2010

              Usage :: ROBOCOPY source destination [file [file]...] [options]

             source :: Source Directory (drive:\path or \\server\share\path).
        destination :: Destination Dir  (drive:\path or \\server\share\path).
               file :: File(s) to copy  (names/wildcards: default is "*.*").

::
:: Copy options :
::
                 /S :: copy Subdirectories, but not empty ones.
                 /E :: copy subdirectories, including Empty ones.
             /LEV:n :: only copy the top n LEVels of the source directory tree.

                 /Z :: copy files in restartable mode.
                 /B :: copy files in Backup mode.
                /ZB :: use restartable mode; if access denied use Backup mode.
            /EFSRAW :: copy all encrypted files in EFS RAW mode.

  /COPY:copyflag[s] :: what to COPY for files (default is /COPY:DAT).
                       (copyflags : D=Data, A=Attributes, T=Timestamps).
                       (S=Security=NTFS ACLs, O=Owner info, U=aUditing info).

           /DCOPY:T :: COPY Directory Timestamps.

               /SEC :: copy files with SECurity (equivalent to /COPY:DATS).
           /COPYALL :: COPY ALL file info (equivalent to /COPY:DATSOU).
            /NOCOPY :: COPY NO file info (useful with /PURGE).

            /SECFIX :: FIX file SECurity on all files, even skipped files.
            /TIMFIX :: FIX file TIMes on all files, even skipped files.

             /PURGE :: delete dest files/dirs that no longer exist in source.
               /MIR :: MIRror a directory tree (equivalent to /E plus /PURGE).

               /MOV :: MOVe files (delete from source after copying).
              /MOVE :: MOVE files AND dirs (delete from source after copying).

     /A+:[RASHCNET] :: add the given Attributes to copied files.
     /A-:[RASHCNET] :: remove the given Attributes from copied files.

            /CREATE :: CREATE directory tree and zero-length files only.
               /FAT :: create destination files using 8.3 FAT file names only.
               /256 :: turn off very long path (> 256 characters) support.

             /MON:n :: MONitor source; run again when more than n changes seen.
             /MOT:m :: MOnitor source; run again in m minutes Time, if changed.

      /RH:hhmm-hhmm :: Run Hours - times when new copies may be started.
                /PF :: check run hours on a Per File (not per pass) basis.

             /IPG:n :: Inter-Packet Gap (ms), to free bandwidth on slow lines.

                /SL :: copy symbolic links versus the target.

            /MT[:n] :: Do multi-threaded copies with n threads (default 8).
                       n must be at least 1 and not greater than 128.
                       This option is incompatible with the /IPG and /EFSRAW options.
                       Redirect output using /LOG option for better performance.


::
:: File Selection Options :
::
                 /A :: copy only files with the Archive attribute set.
                 /M :: copy only files with the Archive attribute and reset it.
    /IA:[RASHCNETO] :: Include only files with any of the given Attributes set.
    /XA:[RASHCNETO] :: eXclude files with any of the given Attributes set.

 /XF file [file]... :: eXclude Files matching given names/paths/wildcards.
 /XD dirs [dirs]... :: eXclude Directories matching given names/paths.

                /XC :: eXclude Changed files.
                /XN :: eXclude Newer files.
                /XO :: eXclude Older files.
                /XX :: eXclude eXtra files and directories.
                /XL :: eXclude Lonely files and directories.
                /IS :: Include Same files.
                /IT :: Include Tweaked files.

             /MAX:n :: MAXimum file size - exclude files bigger than n bytes.
             /MIN:n :: MINimum file size - exclude files smaller than n bytes.

          /MAXAGE:n :: MAXimum file AGE - exclude files older than n days/date.
          /MINAGE:n :: MINimum file AGE - exclude files newer than n days/date.
          /MAXLAD:n :: MAXimum Last Access Date - exclude files unused since n.
          /MINLAD:n :: MINimum Last Access Date - exclude files used since n.
                       (If n < 1900 then n = n days, else n = YYYYMMDD date).

                /XJ :: eXclude Junction points. (normally included by default).

               /FFT :: assume FAT File Times (2-second granularity).
               /DST :: compensate for one-hour DST time differences.

               /XJD :: eXclude Junction points for Directories.
               /XJF :: eXclude Junction points for Files.

::
:: Retry Options :
::
               /R:n :: number of Retries on failed copies: default 1 million.
               /W:n :: Wait time between retries: default is 30 seconds.

               /REG :: Save /R:n and /W:n in the Registry as default settings.

               /TBD :: wait for sharenames To Be Defined (retry error 67).

::
:: Logging Options :
::
                 /L :: List only - don't copy, timestamp or delete any files.
                 /X :: report all eXtra files, not just those selected.
                 /V :: produce Verbose output, showing skipped files.
                /TS :: include source file Time Stamps in the output.
                /FP :: include Full Pathname of files in the output.
             /BYTES :: Print sizes as bytes.

                /NS :: No Size - don't log file sizes.
                /NC :: No Class - don't log file classes.
               /NFL :: No File List - don't log file names.
               /NDL :: No Directory List - don't log directory names.

                /NP :: No Progress - don't display percentage copied.
               /ETA :: show Estimated Time of Arrival of copied files.

          /LOG:file :: output status to LOG file (overwrite existing log).
         /LOG+:file :: output status to LOG file (append to existing log).

       /UNILOG:file :: output status to LOG file as UNICODE (overwrite existinglog).
      /UNILOG+:file :: output status to LOG file as UNICODE (append to existinglog).

               /TEE :: output to console window, as well as the log file.

               /NJH :: No Job Header.
               /NJS :: No Job Summary.

           /UNICODE :: output status as UNICODE.

::
:: Job Options :
::
       /JOB:jobname :: take parameters from the named JOB file.
      /SAVE:jobname :: SAVE parameters to the named job file
              /QUIT :: QUIT after processing command line (to view parameters).

              /NOSD :: NO Source Directory is specified.
              /NODD :: NO Destination Directory is specified.
                /IF :: Include the following Files.
"The democracy will cease to exist when you take away from those who are willing to work and give to those who would not."
-Thomas Jefferson
davodavo
Legit Little One
Legit Little One
Posts: 3
Joined: Wed Mar 17, 2010 11:54 pm

Re: Using Robocopy for backups

Post by davodavo »

Hi. Having a weird and repeated problem with Robocopy, running it on winXP-SP3 with all the latest patches. This problem has been here for a while.

I'm copying files from an NTFS EFS partition to a FAT32 partition. It copies everything fine (although, of course, the encryption is not preserved in the FAT32 partition...which is how I want it).

Now for the problem. When I do a backup, almost all the files are viewed as needing to be refreshed on the FAT32 partition, even though the files haven't been modified at all. If I do a second pass of Robocopy right after doing a backup, only a few dozen files will be re-copied (even though not changed). But if I wait a week, maybe 20% of them will "need" to be copied. If I wait a couple of months, 90% of them "need" to be recopied.

I suspect there's something funny that happens with EFS every time a file is touched at all (e.g., read), and maybe my Virus checker or Windows Defender scans are what is causing this (by reading).

Anybody know of a remedy -- some subtle switch in Robocopy that will pay strict attention to the standard file system modified date (even though these are all EFS files). NOTE: this *only* happens on EFS files, not unencrypted ones.
User avatar
skier
Moderator
Moderator
Posts: 4450
Joined: Mon Mar 26, 2007 3:29 pm
Location: Maine
Contact:

Re: Using Robocopy for backups

Post by skier »

im backing up the entire C directory, with flags: /E /SEC /COPYALL /B /LOG+:ROBOCOPYLOG.TXT /MOT:360 /XO

but it didn't copy the program files folder, windows folder or the users docs/setts folder, just the public one, so i just have the files i dont need, and none of the files i do need lol

i assume this is a permissions problem?
-Austin
Image
Screamin' BCLK: Image
775 System (Overclocking Platform): Q8400/Q8300/E8400/E7400/E7500 - GA-EP45-UD3R v1.1 - 4GB (2x2) OCZ Reaper HPC DDR2 1066 CL5 2.1v Corsair TX-750w
Gamer: Asrock Z77 Extreme4, i7 3770K @4.6GHz, ThermalTake Armor A90 modded, 2x4GB GSKILL RipjawsX DDR3 2133 CL9, Corsair HX-750w, MSI GTX660 Twin Frozr
Server2012: Q9300 - 8GB DDR2 - Asus P5QL Pro - Corsair CX430 - Mirrored 2TB Seagate's with 2TB WD cav for fileshare backups, 1TB WD for OS backups
User avatar
bubba
Staff Writer
Staff Writer
Posts: 4765
Joined: Sun May 01, 2005 10:24 am
Location: STL

Re: Using Robocopy for backups

Post by bubba »

hmm... that is strange. Give it admin rights and see if that works.

why copy the prog and win folders?
"The democracy will cease to exist when you take away from those who are willing to work and give to those who would not."
-Thomas Jefferson
User avatar
skier
Moderator
Moderator
Posts: 4450
Joined: Mon Mar 26, 2007 3:29 pm
Location: Maine
Contact:

Re: Using Robocopy for backups

Post by skier »

so i make sure it holds the outlook files (so all the emails are saved) (and no, i dont personally use outlook because i hate it in every way, parents do)
-Austin
Image
Screamin' BCLK: Image
775 System (Overclocking Platform): Q8400/Q8300/E8400/E7400/E7500 - GA-EP45-UD3R v1.1 - 4GB (2x2) OCZ Reaper HPC DDR2 1066 CL5 2.1v Corsair TX-750w
Gamer: Asrock Z77 Extreme4, i7 3770K @4.6GHz, ThermalTake Armor A90 modded, 2x4GB GSKILL RipjawsX DDR3 2133 CL9, Corsair HX-750w, MSI GTX660 Twin Frozr
Server2012: Q9300 - 8GB DDR2 - Asus P5QL Pro - Corsair CX430 - Mirrored 2TB Seagate's with 2TB WD cav for fileshare backups, 1TB WD for OS backups
User avatar
bubba
Staff Writer
Staff Writer
Posts: 4765
Joined: Sun May 01, 2005 10:24 am
Location: STL

Re: Using Robocopy for backups

Post by bubba »

Testing this now and looks to be working. Has to be run as Admin in Win7

Code: Select all

robocopy C:\ f:\c_back /E /copyall /MIR /R:0 /LOG:test.txt
"The democracy will cease to exist when you take away from those who are willing to work and give to those who would not."
-Thomas Jefferson
User avatar
skier
Moderator
Moderator
Posts: 4450
Joined: Mon Mar 26, 2007 3:29 pm
Location: Maine
Contact:

Re: Using Robocopy for backups

Post by skier »

that appears to be working, just put what you had and added eXclude Older and the 6hr schedule
-Austin
Image
Screamin' BCLK: Image
775 System (Overclocking Platform): Q8400/Q8300/E8400/E7400/E7500 - GA-EP45-UD3R v1.1 - 4GB (2x2) OCZ Reaper HPC DDR2 1066 CL5 2.1v Corsair TX-750w
Gamer: Asrock Z77 Extreme4, i7 3770K @4.6GHz, ThermalTake Armor A90 modded, 2x4GB GSKILL RipjawsX DDR3 2133 CL9, Corsair HX-750w, MSI GTX660 Twin Frozr
Server2012: Q9300 - 8GB DDR2 - Asus P5QL Pro - Corsair CX430 - Mirrored 2TB Seagate's with 2TB WD cav for fileshare backups, 1TB WD for OS backups
User avatar
bubba
Staff Writer
Staff Writer
Posts: 4765
Joined: Sun May 01, 2005 10:24 am
Location: STL

Re: Using Robocopy for backups

Post by bubba »

If they are using outlook you can change where it puts the PST file. Change it to the their MyDoc folder then only backup that.

If its Outlook Express, then those files are stored in the Doc and Settings folder in XP, not sure where win live mail puts stuff.

my test just finished, here is the error I got

"ERROR : You do not have the Manage Auditing user right."

I have never tried to copy prog or win folders before... have to look into this one..
"The democracy will cease to exist when you take away from those who are willing to work and give to those who would not."
-Thomas Jefferson
User avatar
skier
Moderator
Moderator
Posts: 4450
Joined: Mon Mar 26, 2007 3:29 pm
Location: Maine
Contact:

Re: Using Robocopy for backups

Post by skier »

well 6GB/9.1GB copied, but the files are hidden and i can't access them...
-Austin
Image
Screamin' BCLK: Image
775 System (Overclocking Platform): Q8400/Q8300/E8400/E7400/E7500 - GA-EP45-UD3R v1.1 - 4GB (2x2) OCZ Reaper HPC DDR2 1066 CL5 2.1v Corsair TX-750w
Gamer: Asrock Z77 Extreme4, i7 3770K @4.6GHz, ThermalTake Armor A90 modded, 2x4GB GSKILL RipjawsX DDR3 2133 CL9, Corsair HX-750w, MSI GTX660 Twin Frozr
Server2012: Q9300 - 8GB DDR2 - Asus P5QL Pro - Corsair CX430 - Mirrored 2TB Seagate's with 2TB WD cav for fileshare backups, 1TB WD for OS backups
davodavo
Legit Little One
Legit Little One
Posts: 3
Joined: Wed Mar 17, 2010 11:54 pm

Re: Using Robocopy for backups

Post by davodavo »

Anybody have any input / wisdom on this question from a while ago????
davodavo wrote:Hi. Having a weird and repeated problem with Robocopy, running it on winXP-SP3 with all the latest patches. This problem has been here for a while.

I'm copying files from an NTFS EFS partition to a FAT32 partition. It copies everything fine (although, of course, the encryption is not preserved in the FAT32 partition...which is how I want it).

Now for the problem. When I do a backup, almost all the files are viewed as needing to be refreshed on the FAT32 partition, even though the files haven't been modified at all. If I do a second pass of Robocopy right after doing a backup, only a few dozen files will be re-copied (even though not changed). But if I wait a week, maybe 20% of them will "need" to be copied. If I wait a couple of months, 90% of them "need" to be recopied.

I suspect there's something funny that happens with EFS every time a file is touched at all (e.g., read), and maybe my Virus checker or Windows Defender scans are what is causing this (by reading).

Anybody know of a remedy -- some subtle switch in Robocopy that will pay strict attention to the standard file system modified date (even though these are all EFS files). NOTE: this *only* happens on EFS files, not unencrypted ones.
User avatar
bubba
Staff Writer
Staff Writer
Posts: 4765
Joined: Sun May 01, 2005 10:24 am
Location: STL

Re: Using Robocopy for backups

Post by bubba »

only thing I can think of is that the files are being changed somehow due to the encryption. Not sure why they would be updating.
"The democracy will cease to exist when you take away from those who are willing to work and give to those who would not."
-Thomas Jefferson
User avatar
skier
Moderator
Moderator
Posts: 4450
Joined: Mon Mar 26, 2007 3:29 pm
Location: Maine
Contact:

Re: Using Robocopy for backups

Post by skier »

@me, there is a GUI for robocopy, and i used that without a problem (aside from making two copies of everything in backwards directories-i'll figure that out sometime, prolly just from me making manual copies)

the UI is actually pretty good:
Robocopy GUI
-Austin
Image
Screamin' BCLK: Image
775 System (Overclocking Platform): Q8400/Q8300/E8400/E7400/E7500 - GA-EP45-UD3R v1.1 - 4GB (2x2) OCZ Reaper HPC DDR2 1066 CL5 2.1v Corsair TX-750w
Gamer: Asrock Z77 Extreme4, i7 3770K @4.6GHz, ThermalTake Armor A90 modded, 2x4GB GSKILL RipjawsX DDR3 2133 CL9, Corsair HX-750w, MSI GTX660 Twin Frozr
Server2012: Q9300 - 8GB DDR2 - Asus P5QL Pro - Corsair CX430 - Mirrored 2TB Seagate's with 2TB WD cav for fileshare backups, 1TB WD for OS backups
User avatar
bubba
Staff Writer
Staff Writer
Posts: 4765
Joined: Sun May 01, 2005 10:24 am
Location: STL

Re: Using Robocopy for backups

Post by bubba »

See now I have never got that to work LOL

If you want to sergically get only the important files can make a large batch file to get it. On each of the SET_source line to point it at the loc of the files you want. Repeat the block of code for each folder.

this way you can place it on a schedule. Still figuring out to get windows 7 to allow copy of system files in use.

Code: Select all

SETLOCAL
SET _source=”C:\Documents and Settings\printer\Local Settings\Application Data\Identities\{61D38EC9-BFE9-47EC-A286-F94B8C62BF59}\Microsoft\Outlook Express”
SET _dest=”f:\backup\email”
SET _what=/E
:: /COPYALL :: COPY ALL file info
:: /B :: copy files in Backup mode.
:: /SEC :: copy files with SECurity
:: /MIR :: MIRror a directory tree
SET _options=/R:0 /W:0 /LOG:c:\batch\logs\email.txt /NFL /NDL
:: /R:n :: number of Retries
:: /W:n :: Wait time between retries
:: /LOG :: Output log file
:: /NFL :: No file logging
:: /NDL :: No dir logging
ROBOCOPY %_source% %_dest% %_what% %_options%
"The democracy will cease to exist when you take away from those who are willing to work and give to those who would not."
-Thomas Jefferson
dannydan
Legit Little One
Legit Little One
Posts: 1
Joined: Wed Feb 07, 2018 2:10 am

Re: Using Robocopy for backups

Post by dannydan »

I guess backing up all the files requires you to run backups every day/week/month, right? But Robocopy doesn't copy locked/open files and also doesn't copy the timestamps. For that I use GS RichCopy Enterprise which has all the features of RoboCopy along with these two..give it a try!
User avatar
bubba
Staff Writer
Staff Writer
Posts: 4765
Joined: Sun May 01, 2005 10:24 am
Location: STL

Re: Using Robocopy for backups

Post by bubba »

Shouldn't copy open files anyway. Thats how files get corrupted. Robo does do security if you turn the /SEC switch on. You can set a Robo script to run on a schedule.
"The democracy will cease to exist when you take away from those who are willing to work and give to those who would not."
-Thomas Jefferson
Post Reply