librelist archives

« back to archive

Hanging with lots of keepalives

Hanging with lots of keepalives

From:
Felix Almeida
Date:
2015-02-26 @ 15:29
Hi,

  I'm starting to use paramiko (with fabric) to automate many tasks we 
have here (over 3,000 UNIX servers) and it looks promising, however 
sometimes I'm bumping into one annoying problem that I hope someone here 
can help me.

  For some servers (and not all the time) whenever I try to remotely run a
command (e.g. uname) the connections hangs. Turning on debugging I see 
this:

2015/02/26 10:11:48 - DEBUG - starting thread (client mode): 0x987b9090L
2015/02/26 10:11:48 - INFO - Connected (version 2.0, client OpenSSH_6.0)
2015/02/26 10:11:48 - DEBUG - kex 
algos:[u'diffie-hellman-group-exchange-sha256', 
u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', 
u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa', u'ssh-dss'] client 
encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', 
u'arcfour128', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', 
u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', 
u'rijndael-cbc@lysator.liu.se'] server encrypt:[u'aes128-ctr', 
u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', u'aes128-cbc',
u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', 
u'aes256-cbc', u'arcfour', u'rijndael-cbc@lysator.liu.se'] client 
mac:[u'hmac-md5', u'hmac-sha1', u'umac-64@openssh.com', u'hmac-sha2-256', 
u'hmac-sha2-256-96', u'hmac-sha2-512', u'hmac-sha2-512-96', 
u'hmac-ripemd160', u'hmac-ripemd160@openssh.com', u'hmac-sha1-96', 
u'hmac-md5-96'] server mac:[u'hmac-md5', u'hmac-sha1', 
u'umac-64@openssh.com', u'hmac-sha2-256', u'hmac-sha2-256-96', 
u'hmac-sha2-512', u'hmac-sha2-512-96', u'hmac-ripemd160', 
u'hmac-ripemd160@openssh.com', u'hmac-sha1-96', u'hmac-md5-96'] client 
compress:[u'none'] server compress:[u'none'] client lang:[u''] server 
lang:[u''] kex follows?False
2015/02/26 10:11:48 - DEBUG - Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
2015/02/26 10:11:48 - DEBUG - using kex diffie-hellman-group1-sha1; server
key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local 
hmac-sha1, remote hmac-sha1; compression: local none, remote none
2015/02/26 10:11:48 - DEBUG - Switch to new keys ...
2015/02/26 10:11:48 - DEBUG - Adding ssh-rsa host key for myhost: fc0a75...
2015/02/26 10:11:48 - DEBUG - Trying SSH agent key c469c...
2015/02/26 10:11:49 - DEBUG - userauth is OK
2015/02/26 10:11:49 - INFO - Authentication (publickey) successful!
2015/02/26 10:11:49 - DEBUG - [chan 1] Max packet in: 34816 bytes
2015/02/26 10:11:49 - DEBUG - [chan 1] Max packet out: 32768 bytes
2015/02/26 10:11:49 - INFO - Secsh channel 1 opened.
2015/02/26 10:11:49 - DEBUG - [chan 1] Sesch channel 1 request ok
2015/02/26 10:11:52 - DEBUG - Sending global request "keepalive@lag.net"
2015/02/26 10:11:55 - DEBUG - Sending global request "keepalive@lag.net"
2015/02/26 10:11:58 - DEBUG - Sending global request "keepalive@lag.net"
2015/02/26 10:12:01 - DEBUG - Sending global request "keepalive@lag.net"
...

  And these "Sending global request keepalive" messages keeping coming 
forever until I kill the connection. :(

  Any ideas of what am I doing wrong or how can I avoid this issue?

Thanks!!
Felix

PS: I'm running Paramiko 1.14.0 on Python 2.7.8, and Fabric 1.9.1.





________________________________
This communication is confidential. We only send and receive email on the 
basis of the terms set out at 
www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et r?ception de courriels 
se fait strictement suivant les modalit?s ?nonc?es dans l'avis publi? ? 
www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________

Re: Hanging with lots of keepalives

From:
Felix Almeida
Date:
2015-02-26 @ 16:07
Ok, I've found a workaround in case someone is facing the same issue: 
don't allocate a pty.

I'm not sure, but it might be the same issue described here: 
http://stackoverflow.com/questions/24070445/get-pty-via-sshclient-sometimes-hangs-forever-with-paramiko

Apparently fabric guys already knew that (shame on me): 
http://www.fabfile.org/faq.html#init-scripts-don-t-work

Thanks anyway!


From: Felix Almeida
Sent: Thursday, February 26, 2015 10:30 AM
To: 'paramiko@librelist.com'
Subject: Hanging with lots of keepalives

Hi,

  I'm starting to use paramiko (with fabric) to automate many tasks we 
have here (over 3,000 UNIX servers) and it looks promising, however 
sometimes I'm bumping into one annoying problem that I hope someone here 
can help me.

  For some servers (and not all the time) whenever I try to remotely run a
command (e.g. uname) the connections hangs. Turning on debugging I see 
this:

2015/02/26 10:11:48 - DEBUG - starting thread (client mode): 0x987b9090L
2015/02/26 10:11:48 - INFO - Connected (version 2.0, client OpenSSH_6.0)
2015/02/26 10:11:48 - DEBUG - kex 
algos:[u'diffie-hellman-group-exchange-sha256', 
u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', 
u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa', u'ssh-dss'] client 
encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', 
u'arcfour128', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', 
u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', 
u'rijndael-cbc@lysator.liu.se'] server encrypt:[u'aes128-ctr', 
u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', u'aes128-cbc',
u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', 
u'aes256-cbc', u'arcfour', u'rijndael-cbc@lysator.liu.se'] client 
mac:[u'hmac-md5', u'hmac-sha1', u'umac-64@openssh.com', u'hmac-sha2-256', 
u'hmac-sha2-256-96', u'hmac-sha2-512', u'hmac-sha2-512-96', 
u'hmac-ripemd160', u'hmac-ripemd160@openssh.com', u'hmac-sha1-96', 
u'hmac-md5-96'] server mac:[u'hmac-md5', u'hmac-sha1', 
u'umac-64@openssh.com', u'hmac-sha2-256', u'hmac-sha2-256-96', 
u'hmac-sha2-512', u'hmac-sha2-512-96', u'hmac-ripemd160', 
u'hmac-ripemd160@openssh.com', u'hmac-sha1-96', u'hmac-md5-96'] client 
compress:[u'none'] server compress:[u'none'] client lang:[u''] server 
lang:[u''] kex follows?False
2015/02/26 10:11:48 - DEBUG - Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
2015/02/26 10:11:48 - DEBUG - using kex diffie-hellman-group1-sha1; server
key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local 
hmac-sha1, remote hmac-sha1; compression: local none, remote none
2015/02/26 10:11:48 - DEBUG - Switch to new keys ...
2015/02/26 10:11:48 - DEBUG - Adding ssh-rsa host key for myhost: fc0a75...
2015/02/26 10:11:48 - DEBUG - Trying SSH agent key c469c...
2015/02/26 10:11:49 - DEBUG - userauth is OK
2015/02/26 10:11:49 - INFO - Authentication (publickey) successful!
2015/02/26 10:11:49 - DEBUG - [chan 1] Max packet in: 34816 bytes
2015/02/26 10:11:49 - DEBUG - [chan 1] Max packet out: 32768 bytes
2015/02/26 10:11:49 - INFO - Secsh channel 1 opened.
2015/02/26 10:11:49 - DEBUG - [chan 1] Sesch channel 1 request ok
2015/02/26 10:11:52 - DEBUG - Sending global request 
"keepalive@lag.net<mailto:keepalive@lag.net>"
2015/02/26 10:11:55 - DEBUG - Sending global request 
"keepalive@lag.net<mailto:keepalive@lag.net>"
2015/02/26 10:11:58 - DEBUG - Sending global request 
"keepalive@lag.net<mailto:keepalive@lag.net>"
2015/02/26 10:12:01 - DEBUG - Sending global request 
"keepalive@lag.net<mailto:keepalive@lag.net>"
...

  And these "Sending global request keepalive" messages keeping coming 
forever until I kill the connection. :(

  Any ideas of what am I doing wrong or how can I avoid this issue?

Thanks!!
Felix

PS: I'm running Paramiko 1.14.0 on Python 2.7.8, and Fabric 1.9.1.





________________________________
This communication is confidential. We only send and receive email on the 
basis of the terms set out at 
www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et r?ception de courriels 
se fait strictement suivant les modalit?s ?nonc?es dans l'avis publi? ? 
www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________