jan on ruby

once upon a time, there was a bug in the (apart from the bug awesome) Ruby Net::SFTP library that caused the upload of larger than tiny files to freeze. the only thing i was able to find online was in this Capistano group thread where Stuart Sierra says:“Another problem, however: cap freezes while uploading the gzipped
copy of the code. The last thing cap prints to the screen is:

** uploading data to mysite.com:/tmp/20070816195756.tar.gz

Then it does nothing until I kill it. If I log in to the server,
there’s a file at /tmp/20070816195756.tar.gz with zero bytes. Could
this be some obscure bug in Ruby’s SFTP library?”

and Jamis Buck (author of Net::SFTP) replies:

“Sadly, yeah, it sounds like you’re one of the lucky few who are
getting bitten by that one. I’ve never been able to duplicate it
reliably, so I’m not sure what’s causing it, but I’m hoping my rewrite
of Net::SFTP will fix it. (Should have that ready to release by the
end of the month).
In the meantime, I believe some on this list have written custom
strategies to work around that bug… anyone want to chime in here?”

oh and that was in on Aug 16, 2007 so the rewrite might not be here for a while (this is not a complaint, i’m very happy for what there is now) and no one chimed in so i don’t know what the other people’s workarounds might be, but i’ll post mine. first to the bug though:

this small script demonstrates the problem:

require 'rubygems'

require 'net/sftp'

require 'net/http'

uri = URI.parse('http://127.0.0.1')

Net::HTTP.start(uri.host, uri.port) do |ht|

  ht.request_get('/')

end

Net::SFTP.start('faraway.com', 'uploader', 'uploader', :verbose => :info) do |session|

  session.put_file('a.wav', 'a')

end

the upload freezes only when uploading to some machines, so when i replace faraway with another computer (doesn’t have to be localhost), it works. also when i don’t do the GET request first, it uploads the file even to faraway without problems. so this is some kind of concurrency problem that occurs only sometimes. i want to try to look around the source of the library, but haven’t done that yet. first the possible workarounds:

1. if you want to have some real fun, try to separate the upload part of your program to a separate script and call it from your program. you might end up with some novel way of how to pass binary data between ruby scripts like i did.

2. ok now seriously. what turned out to be working is tunneling. so if you set up a tunnel from localhost to faraway and you use Net::SFTP to connect to localhost, it’ll upload the stuff to faraway just fine.
i thought it’d be nice if these tunnels were set up dynamically by the program and closed before it finishes, but i’m not sure how to do that, how to fork() processes in ruby and all that. you might.

3. what i ended up doing was not using the put_file method at all. i use Net::SFTP for creating directories, but for the actual upload… wait for it..:

scp_command = "scp -oIdentityFile=#{private_key} #{local_name} #{user_name}@#{host_name}:#{remote_name}"

@logger.info "Running the following command: #{scp_command}"

unless system("#{scp_command}")

  raise "SCP command failed"

end

this actually makes the transfer at least 4 times faster..

you might need to upload to a server that only allows you to use SFTP and not SSH. i thought i was screwed because i couldn’t install the public key there (without a password-less key you can’t automate SCP commands) but then hey:

sftp> mkdir .ssh

sftp> chmod 700 .ssh

Changing mode on /home/uploader/.ssh

:)