The Demented Ravings of Frank W. Zammetti Visit www.zammetti.com for all things me

18Jan/15Off

Building a homemade Dropbox/Time Machine thingy for vanilla file systems

So, I've had a dream for some time... but before I get to that, let me give you some necessary background...

I've got a server at home, and on that server are various directories (as servers tend to have!). One of them is a directory full of assorted documents. These documents are edited by myself and other members of my house. Documents are created, deleted, sub directories created and deleted, things renamed, all the typical file system activities.

Now, my server is well-protected in terms of backups: the server has a nice RAID array in it for starters, with real-time monitoring and reporting rolled into it. I've got Mozy running, so I have off-site backups covered. Plus, Mozy can do backups to an external hard drive at the same time, so I do that as well for an added layer of security. On top of all of that, every two weeks I image the entire data array to a different external hard drive. If that wasn't enough: every month I burn the most critical stuff to a couple of Blu-Ray discs, and once or twice a year I ship a copy off to my mom in another state.

In other words:

  • I'm incredibly anal about data retention (yes, I've been burned before)
  • I'm really not very likely to lose anything of value off that server

But, here's the thing: all that backup stuff doesn't do one significant thing for me and that's versioning. That's where that dream I mentioned comes into play. You see, I've also got a Subversion server running. What I'd really love to have happen is for at least certain directories on the server be checked into Subversion. That's easy to do of course, but the problem arises when you try and tell your wife or kids how to work with version control. When you aren't used to that thought process it can be pretty foreign. Besides, even if you're used to it, do you really want to have to worry about checking in changes every time you edit a file? Because even I don't really want to.

So... wouldn't it be great if there was a way I could have the server monitor the directory for changes, and when any are seen, automatically check the changes into Subversion?

Yes, it would be great... no, check that: it IS great, because I BUILT IT!

The title of this post is courtesy of a Twitter buddy of mine, Nick Carter (@thynctank). When I was explaining what I was doing, that was how he described it back to me and when I thought about it, it really was a very accurate description!

Ok, fine, preliminaries done, let's get to brass tacks! How did I pull this off?

First, let me say that this is a Windows server. If you're running a *nix variant then you're on your own. For Windows though, check out the app Directory Monitor from DevEnterprise.NET at https://www.deventerprise.net. It's the first part of the equation, namely (and unsurprisingly, given it's name) the directory monitoring. This app is fantastic... it's highly configurable and Just Works™. It's also exactly the ticket.

The next part of the equation is having an Subversion client installed. I loves me some TortoiseSVN (http://tortoisesvn.net). If you're not familiar with it, it's a GUI client for Subversion that integrates with Windows Explorer itself, so you interact with it via context menu entries on files and directories. Fortunately, it also ships with a command line client and that's what we need here. In theory you can use whatever client you want, so long as it's in your path.

The final part of the equation is a VB script that handles the Subversion interactions. That's the bit I wrote and it's kind of the engine of the whole thing. Here's that script:

' **********************************************************************************************************************
' Automated Subversion check-in script by Frank W. Zammetti.
'
' This script is intended to be used with the Directory Monitor application from DevEnterprise.NET
' (https://www.deventerprise.net).  The goal is to real-time monitor a directory for changes and when they are detected,
' execute this script to automatically check all changes into Subversion.
'
' Notes:
'		1. It is assumed that the directory being monitored is a proper working copy of a directory already under
'			 SVN control.
'		2. Directory Monitor must be started with admin privs or the file writes in this script may fail.
'		3. You should be able to monitor multiple directories and handle changes with this same script, however, since the
'			 file writes use the directory of the script itself as the target for file writes, each directory you wish to
'			 monitor should have its own copy of this script and it should be in a different directory from all others.
'		4. This script assumes that it is passed a command line argument that is the fully-qualified path to the directory
'			 being monitored.  Without that, this will all fail horribly.
'		5. There's no error checking throughout this script... partly because there's not a heck of a lot that seems to be
'			 possible in the first place, and partly because what may be possible wouldn't seem to buy us much anyway.
'
' For more information on using this script, see my blog post here:
' http://www.zammetti.com/blog/2015/01/18/building-a-homemade-dropboxtime-machine-thingy-for-vanilla-file-systems
' **********************************************************************************************************************


' Create objects we'll need throughout.
Set oShell = WScript.CreateObject("WScript.Shell")
Set oFSO = CreateObject("Scripting.FileSystemObject")

' If script is already running, abort.  This shouldn't really be necessary because Directory Monitor should be
' configured to not allow concurrent executions, but it's a little added safety net.
If oFSO.FileExists("running.txt") Then
	WScript.Quit
End If

' Determine the path of this script.  This is needed to properly target our file writes.
sScriptPath = Wscript.ScriptFullName
Set oScriptFile = oFSO.GetFile(Wscript.ScriptFullName)
sScriptPath = oFSO.GetParentFolderName(oScriptFile)

' Write our "running.txt" file to avoid this script running again until this execution is done.
Set oRunningFile = oFSO.CreateTextFile(sScriptPath & "\running.txt")
oRunningFile.Write "running"
oRunningFile.Close

' Start creating our results file.
sResults = "----------------------------------------------------------------------" & VbCrLf
sResults = sResults & "Script starting " & DateInfo & Now & VbCrLf

' Grab the command line argument so we know what directory to work with.
sTargetDir = Wscript.Arguments(0)
sResults = sResults & "Target directory = " & sTargetDir & VbCrLf & VbCrLf

' Get the list of changes in the target directory.
sCmd = "svn status " & Chr(34) & sTargetDir & Chr(34)
sResults = sResults & sCmd & VbCrLf
Set oExecStatus = oShell.Exec(sCmd)

' Only continue once the command completes.
Do While oExecStatus.Status = 0
	WScript.Sleep 100
Loop

' Iterate over the changes and handle each appropriately.
Do

	' Get the next change.
	sLine = oExecStatus.StdOut.ReadLine()
	sResults = sResults & sLine & VbCrLf

	' Determine SVN command to execute.  Note that only adds and deletes require
	' use to do anything, the SVN ci command takes care of modifications.
	sOp = ""
	If Left(sLine, 1) = "?" Then
		sOp = "add"
	ElseIf Left(sLine, 1) = "!" Then
		sOp = "delete"
	End If

	' Execute the command, if there is one to execute.
	If sOp <> "" Then
		sCmd = "svn " & sOp & " " & Chr(34) & Trim(Mid(sLine, 2)) & Chr(34)
		sResults = sResults & sCmd & VbCrLf
		Set oExecOp = oShell.Exec(sCmd)
		' Only continue once the command completes.
		Do While oExecOp.Status = 0
			WScript.Sleep 100
		Loop
	End If

Loop While Not oExecStatus.Stdout.atEndOfStream

' Fire an SVN ci command to commit all changes.
sCmd = "svn ci " & "-m " & Chr(34) & "Auto-Commit" & Chr(34) & " " & Chr(34) & sTargetDir & Chr(34)
sResults = sResults & VbCrLf & sCmd & VbCrLf
Set oExecCommit = oShell.Exec(sCmd)
' Only continue once the command completes.
Do While oExecCommit.Status = 0
	WScript.Sleep 100
Loop
' Capture the output of the command in our results.
Do
	sResults = sResults & oExecCommit.StdOut.ReadLine() & VbCrLf
Loop While Not oExecCommit.Stdout.atEndOfStream

' Epilogue in the results file.
sResults = sResults & vbCrLf & "Script finished " & DateInfo & Now & VbCrLf & VbCrLf

' Write results to file.
iForAppending = 8
Set oResultsFile = oFSO.OpenTextFile(sScriptPath & "\results.txt", iForAppending, True)
oResultsFile.Write sResults
oResultsFile.Close

' Delete our "running.txt" file so this script can run next time it needs to.
oFSO.DeleteFile(sScriptPath & "\running.txt")

Copy this down and save it to a directory (NOT the one you'll be monitoring!) and name it commit.vbs and set it aside for now.

Putting all the pieces of the puzzle together is pretty easy. Let's say we're going to monitor a directory C:\docs. The first step is configuring Directory Monitor. Here's the main Directory Monitor window:

2015-01-18_15-26-30

As you can see, I've already done the configuration and have even had some events triggered during my testing. I'll walk you through the setup screens that got me to that point. If you're adding the directory for the first time you'd be presented with these same screens. Here's the first one:

2015-01-18_15-24-19

This is actually the most important of the bunch (well, one of two key tabs anyway). You'll need to check off the events you see checked here to ensure you capture everything that happens in the directory, but no more. You'll also likely want to monitor sub-directories, but that's up to you. The next critical thing is to ensure you exclude any changes that happen in the .svn directory. Failing to do that will result in an endless loop or script executions when we add in the execution of the script (ask me how I know!).

The Text Log tab can be left alone, but for the sake of completeness, here's what it looks like on my machine:

2015-01-18_15-24-38

The Execute tab is the other key tab, and here it is:

2015-01-18_15-25-29

Browse for the commit.vbs script file and select it in the Execute box.  Directory Monitor should modify it to look like what you see here.  Now, in the Parameters box, add the %dirpath% placeholder.  That will be passed to the script so it knows what directory to work on.

Like the Text Log tab, the Sounds, Emailed and Database tabs don't matter in my use case so I've left them with default setting.  But, just to ensure you don't miss anything, here they are as I see them on my machine:

2015-01-18_15-25-48

2015-01-18_15-26-05

2015-01-18_15-26-16

At this point, hit Save and you're basically done!  Directory Monitor will now start monitoring the directory and will fire the script when any changes occur.  You should see a couple of command prompt windows flash across the screen whenever it does. I'm working on a way to avoid that, but it only winds up mattering if you've got a lot of activity in the directory and need to work on the server. In that case, just pause monitoring in Directory Monitor and you should be fine. During this processing, the results.txt file will be written so you can see what happened.  As an example, here's a couple of runs recorded during my testing:

----------------------------------------------------------------------
Script starting 1/18/2015 3:20:48 PM
Target directory = C:\docs</code>

svn status "C:\docs"
M C:\docs\_Our Movies.txt

svn ci -m "Auto-Commit" "C:\docs"
Sending docs\_Our Movies.txt
Transmitting file data .
Committed revision 5777.

Script finished 1/18/2015 3:20:49 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:20:59 PM
Target directory = C:\docs

svn status "C:\docs"
? C:\docs\test - Copy (2).txt
svn add "C:\docs\test - Copy (2).txt"
add - A C:\docs\test - Copy (2).txt

svn ci -m "Auto-Commit" "C:\docs"
Adding docs\test - Copy (2).txt
Transmitting file data .
Committed revision 5778.

Script finished 1/18/2015 3:21:00 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:21:00 PM
Target directory = C:\docs

svn status "C:\docs"

svn ci -m "Auto-Commit" "C:\docs"

Script finished 1/18/2015 3:21:01 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:21:42 PM
Target directory = C:\docs

svn status "C:\docs"
? C:\docs\test - Copy (3).txt
svn add "C:\docs\test - Copy (3).txt"

svn ci -m "Auto-Commit" "C:\docs"
Adding docs\test - Copy (3).txt
Transmitting file data .
Committed revision 5779.

Script finished 1/18/2015 3:21:43 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:21:43 PM
Target directory = C:\docs

svn status "C:\docs"

svn ci -m "Auto-Commit" "C:\docs"

Script finished 1/18/2015 3:21:44 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:22:05 PM
Target directory = C:\docs

svn status "C:\docs"
! C:\docs\test - Copy (2).txt
svn delete "C:\docs\test - Copy (2).txt"
! C:\docs\test - Copy (3).txt
svn delete "C:\docs\test - Copy (3).txt"
! C:\docs\test - Copy.txt
svn delete "C:\docs\test - Copy.txt"

svn ci -m "Auto-Commit" "C:\docs"
Deleting docs\test - Copy (2).txt
Deleting docs\test - Copy (3).txt
Deleting docs\test - Copy.txt

Committed revision 5780.

Script finished 1/18/2015 3:22:06 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:22:06 PM
Target directory = C:\docs

svn status "C:\docs"

svn ci -m "Auto-Commit" "C:\docs"

Script finished 1/18/2015 3:22:07 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:22:07 PM
Target directory = C:\docs

svn status "C:\docs"

svn ci -m "Auto-Commit" "C:\docs"

Script finished 1/18/2015 3:22:07 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:22:47 PM
Target directory = C:\docs

svn status "C:\docs"
? C:\docs\work
svn add "C:\docs\work"

svn ci -m "Auto-Commit" "C:\docs"
Adding docs\work

Committed revision 5781.

Script finished 1/18/2015 3:22:48 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:22:54 PM
Target directory = C:\docs

svn status "C:\docs"
? C:\docs\work\test.txt
svn add "C:\docs\work\test.txt"

svn ci -m "Auto-Commit" "C:\docs"
Adding docs\work\test.txt
Transmitting file data .
Committed revision 5782.

Script finished 1/18/2015 3:22:55 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:22:56 PM
Target directory = C:\docs

svn status "C:\docs"

svn ci -m "Auto-Commit" "C:\docs"

Script finished 1/18/2015 3:22:56 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:23:15 PM
Target directory = C:\docs

svn status "C:\docs"
? C:\docs\aaa
svn add "C:\docs\aaa"

svn ci -m "Auto-Commit" "C:\docs"
Adding docs\aaa

Committed revision 5783.

Script finished 1/18/2015 3:23:16 PM

----------------------------------------------------------------------
Script starting 1/18/2015 3:23:21 PM
Target directory = C:\docs

svn status "C:\docs"
! C:\docs\aaa
svn delete "C:\docs\aaa"

svn ci -m "Auto-Commit" "C:\docs"
Deleting docs\aaa

Committed revision 5784.

Script finished 1/18/2015 3:23:22 PM

As you can see, this file will just continue to grow unbounded (another thing I intend to remedy later) so if you're going to have a lot of activity in the directory and are worried about space you may want to comment out the lines in the script that write it (at the cost of a loss of logging obviously).

Well, that's basically a wrap! I can't claim any of this is rocket science or anything, at the end of the day it's pretty simple, but it's quite a useful script and achieves my stated dream 🙂 If you find it useful too then by all means, have at it! If you make any enhancements I'd love to hear about them... this thing is running on my server now so I have a vested interest in it working correctly and running well so if it can be improved I'd love to know about it.

Comments (0) Trackbacks (0)

Sorry, the comment form is closed at this time.

Trackbacks are disabled.