Page MenuHomePhabricator

0.8.4 ssh server regression with interactive keyboard auth: message callback only called once
Closed, ResolvedPublic

Description

Cockpit's ssh unit tests started to fail in Fedora Rawhide when libssh got updated from 0.8.3 to 0.8.4 (https://apps.fedoraproject.org/koschei/build/5531857)

In particular this affects interactive keyboard authentication on the ssh *server* side; cockpit's unit tests have a libssh based mock server, which can simulate various scenarios, including a multi-prompt challenge (like password and token). I already established that libssh on the client side still works fine (I ran it against a real SSH server).

The effect is that a client just hangs and eventually times out when doing keyboard interactive authentication, when the server runs against libssh 0.8.4:

$ make run-server
./mock-sshd --user=joe --password=foobar --port 1234 -v
[2018/10/19 16:50:55.036499, 2] ssh_pki_import_privkey_base64:  Trying to decode privkey passphrase=false
[2018/10/19 16:50:56.925774, 1] ssh_server_connection_callback:  SSH client banner: SSH-2.0-libssh_0.8.3
[2018/10/19 16:50:56.925859, 1] ssh_analyze_banner:  Analyzing banner: SSH-2.0-libssh_0.8.3
[2018/10/19 16:50:56.926681, 2] ssh_kex_select_methods:  Negotiated curve25519-sha256,ssh-rsa,aes256-ctr,aes256-ctr,hmac-sha2-256,hmac-sha2-256,none,none,,
[2018/10/19 16:50:56.981046, 2] ssh_server_curve25519_init:  SSH_MSG_KEX_ECDH_REPLY sent
[2018/10/19 16:50:56.981130, 2] ssh_server_curve25519_init:  SSH_MSG_NEWKEYS sent
[2018/10/19 16:50:56.986727, 2] ssh_packet_newkeys:  Received SSH_MSG_NEWKEYS

** (mock-sshd:24621): WARNING **: 16:50:56.987: authenticate_callback: round 0, message type -1

** (mock-sshd:24621): WARNING **: 16:50:56.987: authenticate_callback: round 0, message type -1, deny, ret 1

** (mock-sshd:24621): WARNING **: 16:50:56.987: authenticate_callback: round 0, message type 16

** (mock-sshd:24621): WARNING **: 16:50:56.987: auth_interactive begin: round 0
[2018/10/19 16:50:56.987653, 2] ssh_message_auth_interactive_request:  Warning: Got a keyboard-interactive response but it seems we didn't send the request.

** (mock-sshd:24621): WARNING **: 16:50:56.987: auth_interactive round 1, ret 1

** (mock-sshd:24621): WARNING **: 16:50:56.987: authenticate_callback: round 1, message type 16, more, ret 0
$ make run-client
./client
[2018/10/19 16:50:56.925185, 2] ssh_connect:  libssh 0.8.3 (c) 2003-2018 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
[2018/10/19 16:50:56.925476, 2] ssh_socket_connect:  Nonblocking connection socket: 3
[2018/10/19 16:50:56.925511, 2] ssh_connect:  Socket connecting, now waiting for the callbacks to work
[2018/10/19 16:50:56.925549, 1] socket_callback_connected:  Socket connection callback: 1 (0)
[2018/10/19 16:50:56.925969, 1] ssh_client_connection_callback:  SSH server banner: SSH-2.0-libssh_0.8.4
[2018/10/19 16:50:56.926003, 1] ssh_analyze_banner:  Analyzing banner: SSH-2.0-libssh_0.8.4
[2018/10/19 16:50:56.926548, 2] ssh_kex_select_methods:  Negotiated curve25519-sha256,ssh-rsa,aes256-ctr,aes256-ctr,hmac-sha2-256,hmac-sha2-256,none,none,,
[2018/10/19 16:50:56.981151, 2] ssh_packet_dh_reply:  Received SSH_KEXDH_REPLY
[2018/10/19 16:50:56.986690, 2] ssh_client_curve25519_reply:  SSH_MSG_NEWKEYS sent
[2018/10/19 16:50:56.986736, 2] ssh_packet_newkeys:  Received SSH_MSG_NEWKEYS
[2018/10/19 16:50:56.986995, 2] ssh_packet_newkeys:  Signature verified and valid
kbdint instruction: Password
prompt: Password

Here the message callback (authenticate_callback()) is being called just once, then the client sends a second message, which then never triggers another callback and thus client and server are just waiting on each other.

When running mock-sshd against libssh 0.8.3, a second callback gets fired, and the authentication completes:

$ make run-server
./mock-sshd --user=joe --password=foobar --port 1234 -v
[2018/10/19 16:55:50.803414, 2] ssh_pki_import_privkey_base64:  Trying to decode privkey passphrase=false
[2018/10/19 16:55:52.635496, 1] ssh_server_connection_callback:  SSH client banner: SSH-2.0-libssh_0.8.3
[2018/10/19 16:55:52.635557, 1] ssh_analyze_banner:  Analyzing banner: SSH-2.0-libssh_0.8.3
[2018/10/19 16:55:52.636138, 2] ssh_kex_select_methods:  Negotiated curve25519-sha256,ssh-rsa,aes256-ctr,aes256-ctr,hmac-sha2-256,hmac-sha2-256,none,none,,
[2018/10/19 16:55:52.690456, 2] ssh_server_curve25519_init:  SSH_MSG_KEX_ECDH_REPLY sent
[2018/10/19 16:55:52.690555, 2] ssh_server_curve25519_init:  SSH_MSG_NEWKEYS sent
[2018/10/19 16:55:52.695387, 2] ssh_packet_newkeys:  Received SSH_MSG_NEWKEYS

** (mock-sshd:25085): WARNING **: 16:55:52.695: authenticate_callback: round 0, message type -1

** (mock-sshd:25085): WARNING **: 16:55:52.695: authenticate_callback: round 0, message type -1, deny, ret 1

** (mock-sshd:25085): WARNING **: 16:55:52.696: authenticate_callback: round 0, message type 16

** (mock-sshd:25085): WARNING **: 16:55:52.696: auth_interactive begin: round 0
[2018/10/19 16:55:52.696237, 2] ssh_message_auth_interactive_request:  Warning: Got a keyboard-interactive response but it seems we didn't send the request.

** (mock-sshd:25085): WARNING **: 16:55:52.696: auth_interactive round 1, ret 1

** (mock-sshd:25085): WARNING **: 16:55:52.696: authenticate_callback: round 1, message type 16, more, ret 0

** (mock-sshd:25085): WARNING **: 16:55:52.696: authenticate_callback: round 1, message type 16

** (mock-sshd:25085): WARNING **: 16:55:52.696: auth_interactive begin: round 1

** (mock-sshd:25085): WARNING **: 16:55:52.696: auth_interactive round 1, ret 0

** (mock-sshd:25085): WARNING **: 16:55:52.696: authenticate_callback: round 1, message type 16, accept, ret 0
[2018/10/19 16:55:52.738513, 2] channel_rcv_change_window:  Adding 1216012 bytes to channel (43:43) (from 63988 bytes)
[2018/10/19 16:55:52.738985, 1] ssh_socket_exception_callback:  Socket exception callback: 1 (0)
[2018/10/19 16:55:52.739066, 1] ssh_socket_exception_callback:  Socket error: disconnected
$ make run-client
./client
[2018/10/19 16:56:26.250654, 2] ssh_connect:  libssh 0.8.3 (c) 2003-2018 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
[2018/10/19 16:56:26.250951, 2] ssh_socket_connect:  Nonblocking connection socket: 3
[2018/10/19 16:56:26.250989, 2] ssh_connect:  Socket connecting, now waiting for the callbacks to work
[2018/10/19 16:56:26.251048, 1] socket_callback_connected:  Socket connection callback: 1 (0)
[2018/10/19 16:56:26.251500, 1] ssh_client_connection_callback:  SSH server banner: SSH-2.0-libssh_0.8.3
[2018/10/19 16:56:26.251545, 1] ssh_analyze_banner:  Analyzing banner: SSH-2.0-libssh_0.8.3
[2018/10/19 16:56:26.252098, 2] ssh_kex_select_methods:  Negotiated curve25519-sha256,ssh-rsa,aes256-ctr,aes256-ctr,hmac-sha2-256,hmac-sha2-256,none,none,,
[2018/10/19 16:56:26.307201, 2] ssh_packet_dh_reply:  Received SSH_KEXDH_REPLY
[2018/10/19 16:56:26.312642, 2] ssh_client_curve25519_reply:  SSH_MSG_NEWKEYS sent
[2018/10/19 16:56:26.312687, 2] ssh_packet_newkeys:  Received SSH_MSG_NEWKEYS
[2018/10/19 16:56:26.312914, 2] ssh_packet_newkeys:  Signature verified and valid
kbdint instruction: Password
prompt: Password
[2018/10/19 16:56:26.313759, 2] channel_open:  Creating a channel 43 with 64000 window and 32768 max packet
[2018/10/19 16:56:26.314067, 2] ssh_packet_channel_open_conf:  Received a CHANNEL_OPEN_CONFIRMATION for channel 43:43
[2018/10/19 16:56:26.314115, 2] ssh_packet_channel_open_conf:  Remote window : 32000, maxpacket : 35000
[2018/10/19 16:56:26.314577, 2] channel_request:  Channel request exec success
[2018/10/19 16:56:26.355358, 2] grow_window:  growing window (channel 43:43) to 1280000 bytes
hello world

This also happens when using the standard ssh client. I added that as make run-ssh; when you use that, you have to type the password ("foobar") into the prompt interactively, and it times out after 10 seconds. Thus ./client is more convenient, it noninteractively supplies the password and has the timeout set to one hour so that it's easier to gdb stuff.

So I'm fairly sure the client side (client.c) is implemented correctly. Of course it could be that mock-sshd.c is doing something wrong. It hasn't changed much in many years and has been working well so far. But I can't see an obvious mismatch between what it does and what the documentation says.

I attach the mock-sshd.c, client.c, test keys, and Makefile here, so that it's convenient to run.

Event Timeline

martinpitt added a project: Restricted Project.

reproducer code and script:

asn added a subscriber: asn.Oct 19 2018, 9:57 PM

e5cee205c17ca0f95b270bcf2baa052e46a85ff5 should fix this, could you please test?

I can confirm that with libssh master, both the reproducer and cockpit's unit tests work. Thanks! As this blocks releasing new cockpit versions, and fixes a regression, do you plan to fix this in Fedora soon? I'll file a bug for the next RHEL devel version.

asn added a comment.Oct 22 2018, 6:09 PM

I will do a release in the next days.

zer0 added a subscriber: zer0.Oct 24 2018, 12:06 PM

Hi,

It looks the 0.7 branch does not compile after commit 734e3ce6747a, which is a backport of e5cee205c17c.

Sorry, I'm not sure it is the proper place to post the fix, but here it is:

From bda46b4a7b46b038d7fddd2c0c3db77f47d60a41 Mon Sep 17 00:00:00 2001
From: Olivier Matz <olivier.matz@6wind.com>
Date: Wed, 24 Oct 2018 11:54:04 +0200
Subject: [PATCH] server: fix access to invalid auth field

The 0.7 branch does not have the commit 73c9d60e5abf ("session: Group
auth variables in a struct"), therefore the access to the auth_state
field is different.

Fixes: 734e3ce6747a ("server: Set correct state after sending INFO_REQUEST (Kbd Interactive)")
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
---
 src/server.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/server.c b/src/server.c
index b1f01a86..400b04a7 100644
--- a/src/server.c
+++ b/src/server.c
@@ -976,7 +976,7 @@ int ssh_message_auth_interactive_request(ssh_message msg, const char *name,
     msg->session->kbdint->prompts = NULL;
     msg->session->kbdint->echo = NULL;
   }
-  msg->session->auth.state = SSH_AUTH_STATE_INFO;
+  msg->session->auth_state = SSH_AUTH_STATE_INFO;

   return rc;
 }
--
2.11.0
asn closed this task as Resolved.Oct 29 2018, 2:22 PM
asn claimed this task.

Fixed with libssh 0.8.5 and libssh 0.7.7.