July 18th, 2004

line rate transfers over the LAN

Sometimes I stumble across these things that I hope google archives when I convert them to text, because they are very important to me and hopefully others will find these blurbs I type up and save some time themselves.

Line rate ethernet. What is it?

Ethernet is all done inside a physical chip on the ethernet card called the PHY. (say FIE) The PHY does stuff that's over my head, but I know that crappy ethernet cards can't even do line rate, which right now is either 10 or 100 megabits a second. Let's pretend you have ethernet cards that can do line rate, 100 megabits.

You want to copy your mp3 directory over the lan. Whats the fastest way to do this with linux or some other *nix?

Samba? Hell no.
NFS? That would probably work. I dunno how to use it. Never bothered.
FTP? Annoying as hell but it'll work. You wont go line rate unless you're using some client that actually transverses the directory structure and does the stuff automatically.

Here's my trick of the day.

Netcat. Here's what you can do with netcat: pipe stuff to a port. Oh, goodness I had no idea what that meant until I realized how simple and elegant it is.

Let's say you're using something like tar to create, oh, a tarfile. We all know perfectly well that you can write a file to standard output (and read from standard input, no less!) A lot of the time, I find it easier to just tar a directory into stdout and pipe it to a tar in the right directory reading stdin, and letting the data stream just explode wherever I aim it. I've done this using SSH to pipe the streams together but frankly, SSH has a lot of overhead. Netcat has none of that, so, just use netcat!

Here's how I am moving my 36 gig mp3 directory over the lan, at which I have measured to be 96.0 megabits a second (effectively "line rate" with the other small encodings occuring. in other words, it couldn't be going any faster with this hardware = perfect)

On the machine to which you're sending your illegal mp3s:

cd /some/directory/where/you/want/incoming/mp3s
nc -l -p 1234|tar xf -

ok, leave that. it'll sit there waiting to be contacted on port 1234.

go to your machine with your mp3s.

do this.
cd /your/directory/before/your/mp3/directory
tar cf - mp3dir|nc 1234

Thats it! Now tar will write to a pipe, which on the other side is nc, which is talking over the lan, to another nc, to a pipe, to tar, which writes to disk. How easy is that?! Think of the time you'll save next time you use this cool trick.

note: obviously you dont have to actually be in the directory you needed to be in. you could just cluster that all into one command:

nc -l -p 1234|"(cd /some/place && tar xf -)" on the destination machine
"(cd /mp3s && tar cf -)"|nc 1234 on the source machine, that should also work.

ahhh netcat, probably yet another tool i should have known about 5 years ago.