Tuesday, October 07, 2003

QuickPar 0.7

Here's a nifty little tool. One of the problems with storing data on media like CD-R / DVD-R is that eventually, your media starts to go bad and some of the information on the CD/DVD will not be recoverable. (Even though CD/DVDs implement a low-level version of error correcting codes which allow them to recover from minor errors.)

One solution has been to burn everything twice (mirroring). However, this is a bit wasteful and also has the problem of managing twice as many pieces of media. Instead, there are now programs that work with .par and .par2 files. The one that I'm using is QuickPar 0.7 which runs under Windows 2000/XP (possibly older versions of windows too).

The way QuickPar works is that you select a bunch of files, and then create recovery files (.par2) that are associated with those files. The .par2 files contain information needed to verify the source files, as well as enough duplicate information to recover bad information in the source files, up to a certain limit of damaged bytes. This is all done using Reed Solomon Codes, which is a mathematical way of reconstructing missing or bad data in a data stream.

For instance, you have 4000 files, maybe 1Gb in size and you tell QuickPar to create 10% redundancy for these files. QuickPar will create .par2 files that will allow you to recover any files that get corrupted or are unreadable, as long as the total number of corrupt/missing bytes is less then 10% of the total. Pretty slick and for the paranoids, you can change the amount of redundancy that you want (1% to 100%), depending on how much space you're willing to sacrifice. (I find that the 10% works well for when I have lots of small files... and I use 20% if I only have a few big files.)

One downside is that it takes a while to calculate and create the .par2 files... my 1Ghz machine takes 5 hours to calculate/create .par2 files for source info of 3800 files, 1.1Gb of data (91Mb/sec throughput). Fortunately, QuickPar allows me to run it at "idle" priority, which means I'm able to still use the system for other tasks. Smaller data sets (such as 90Mb of MP3s from a single album) only take 15-45 seconds to calculate/create the .par2 files for.

One suggestion that I would make is that if you're going to put a single 600Mb file onto CD/DVD, you should probably use some sort of file splitter (e.g. Info-Zips "zipsplit" program) and break the file into 10Mb chunks. Usually, when a file goes bad on a CD/DVD, you're unable to read the entire file back - unless you figure out how to trick Windows into reading a partially good file. So if you have a single 600Mb file, you're kinda gonna still be in trouble because you won't be able to pull that single file back off the CD/DVD in order to repair it. By breaking the file into smaller chunks, only the chunks that are bad won't be copieable and QuickPar will be able to fill in the missing gaps by reconstructing the file from the .par2 files.

Both Info-ZIP and QuickPar are open source / open formats... which means you have good odds that you'll always be able to read your data down the road. (QuickPar is becoming very popular on USENET.)

And... a bunch of links regarding "Reed Solomon Error Correction Coding".

Reed-Solomon error correction - Wikipedia definition.

Reed-Solomon Codes and CD Encodin - Describes how ECC is implemented on audio CDs (it's a bit technical... no pun intended).

The Ubiquitous Reed-Solomon Codes - Barry Cipra, SIAM News, Vol. 26, No. 1, January 1993

The Error Correcting Codes (ECC) Site - A site dedicated to ECC.

RSCode (SourceForge) - an older open-source project which implemented RS encoding for general purpose use.

Digital Signal Processing - Discusses RS codes with regards to DSP.

posted by Wuphon's at 5:29 PM

Monday, October 06, 2003

Digital Billboard

BBC NEWS | Technology | Digital display dazzles London: "'It is possible to talk to the computer across the internet and send it imagery or create imagery on it from a remote location that is completely unique,' he told the BBC programme Go Digital."

I'm wondering just how long it will take before some hacker managed to subvert the display for their own use. Talk about a huge target (in more ways then one).

posted by Wuphon's at 12:08 PM

Sunday, October 05, 2003

CoolPlayer

I'm in the process of looking for a new software MP3 player for Windows... WinAmp uses up way too much memory (20Mb) and all of the builds since last December are buggy on my system. (WinAmp 3 has a bad habit of not launching properly from the WinAgent system tray icon.)

So I'm currently testing CoolPlayer which is a small, open-source, media player (memory footprint is 8Mb). There's no installation required, just extract the .zip file to an empty folder in your programs directory.

For a good minimalist skin (I like to run my mp3 players in "always on top mode" so I need a small visual footprint), try slim skin. Again, there's no install program, just create a folder under your CoolPlayer program folder, then go into options and browse to the .ini file location.

posted by Wuphon's at 3:24 PM