OpenSSL's DTLSv1_Listen() pauses program at runtime

I'm trying to test with OpenSSL DTLS by making a program that creates a client and server socket to echo strings between the sockets; However, when I try to test out DTLSv1_Listen() function my program seems to pause even when I am not trying connecting or sending data between the sockets. note: I am using a post 1.0.2 OpenSSL which is after DTLSv1_Listen() was rewritten.

Here is my complete C++ winsock specific code:

#include <stdio.h>
#include <winsock2.h>
#include <ws2tcpip.h>
#include <openssl/err.h>
#include <openssl/ssl.h>
#include <openssl/err.h>
//#include <openssl/applink.c>
#include <string>
#pragma comment(lib, "Ws2_32.lib")

struct DTLSStuff { //struct to contain DTLS object instances
    SSL_CTX *ctx;
    SSL *ssl;
    BIO *bio;

void DTLSErr() { //DTLS error reporting

int newSocket(sockaddr_in addr) { //creates a socket and returns the file descriptor //TODO expand for multi-platform
    WSADATA wsaData;
    int fd;
    int iResult;

    iResult = WSAStartup(MAKEWORD(2, 2), &wsaData);                                                     //Initialize Winsock
    if (iResult != 0) { printf("WSAStartup failed: %d\n", iResult); exit(1); }

    fd = socket(AF_INET, SOCK_DGRAM, 0); if (fd < 0) { perror("Unable to create socket"); exit(1); }    //create socket
    printf("New Socket: %i\n", fd);
    if (bind(fd, (struct sockaddr *)&addr, sizeof(sockaddr)) < 0) { printf("bind failed with error %u\n", WSAGetLastError());   exit(1); }

    return fd;                                                                                          //file descriptor

void InitCTX(SSL_CTX *ctx, bool IsClient) { //Takes a ctx object and initializes it for DTLS communication
    if (IsClient) {
        if(SSL_CTX_use_certificate_chain_file(ctx, "client-cert.pem") < 0) { printf("Failed loading client cert");}
        if(SSL_CTX_use_PrivateKey_file(ctx, "client-key.pem", SSL_FILETYPE_PEM) < 0) { printf("Failed loading client key"); }
    else {
        if (SSL_CTX_use_certificate_chain_file(ctx, "server-cert.pem") < 0) { printf("Failed loading client cert"); }
        if (SSL_CTX_use_PrivateKey_file(ctx, "server-key.pem", SSL_FILETYPE_PEM) < 0) { printf("Failed loading client key"); }
    //SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, verify_cert);    //omitted for testing
    //SSL_CTX_set_cookie_generate_cb(ctx, generate_cookie);     //omitted for testing
    //SSL_CTX_set_cookie_verify_cb(ctx, verify_cookie);         //omitted for testing
    SSL_CTX_set_read_ahead(ctx, 1);

int main() { //creates client and server sockets and DTLS objects. TODO: have client complete handshake with server socket and send a message and have the server echo it back to client socket
    BIO_ADDR *faux_addr = BIO_ADDR_new(); // for DTLSv1_listen(), since we are this is both client and server (meaning client address is known) it is only used to satisfy parameters.

    //Set up addresses
    sockaddr_in client_addr;
    client_addr.sin_family = AF_INET;
    client_addr.sin_port = htons(25501);
    client_addr.sin_addr.s_addr = INADDR_ANY;
    sockaddr_in server_addr;
    server_addr.sin_family = AF_INET;
    server_addr.sin_port = htons(25500);
    server_addr.sin_addr.s_addr = INADDR_ANY;

    DTLSStuff ClientInf;
    ClientInf.ctx = SSL_CTX_new(DTLSv1_client_method());
    int ClientFD = newSocket(client_addr); = BIO_new_dgram(ClientFD, BIO_NOCLOSE);
    ClientInf.ssl = SSL_new(ClientInf.ctx);
    //SSL_set_options(ClientInf.ssl, SSL_OP_COOKIE_EXCHANGE); //omitted for testing

    DTLSStuff ServerInf;
    ServerInf.ctx = SSL_CTX_new(DTLSv1_server_method());
    int ServerFD = newSocket(server_addr); = BIO_new_dgram(ServerFD, BIO_NOCLOSE);
    ServerInf.ssl = SSL_new(ServerInf.ctx);
    //SSL_set_options(ServerInf.ssl, SSL_OP_COOKIE_EXCHANGE); //omitted for testing

    printf("Listen attempt...\n");  
    int ret = DTLSv1_listen(ServerInf.ssl, faux_addr);
    if (ret < 0) { DTLSErr(); }
    printf("this print should occur, but it never does");

I expect the results to be as follow:

NewSocket: 356
NewSocket: 360
Listen attempt...
this print should occur but it never does

However when running the program it never prints the last line. The program seems to respond as I am able to cancel the executable by ctrl+c so I am assuming it has not crashed or froze, but aside from that I am at a loss. My understanding is that the method should return 0 if nothing happens, >1 if it heard a clienthello, and <0 if an error occurred.

Also, a somewhat related question: Since DTLSv1_Listen() requires a BIO_ADDR to store the incoming requests address does that mean that separate client and servers programs will both require 2 sockets if they want to be able to both send and listen? Normally UDP clients and servers only need a single socket, but I cannot seem to figure a design to retain this with OpenSSL's DTLS.

I thank you for your time.

1 answer

  • answered 2017-12-06 01:23 Matt Caswell

    I don't see anywhere in your code where you set the socket to be non-blocking. In the default blocking mode when you attempt to read from the socket your program will pause until data has arrived. If you don't want that then make sure your set the appropriate option (I'm not a Windows programmer, but ioctlsocket seems to do the job:

    does that mean that separate client and servers programs will both require 2 sockets if they want to be able to both send and listen

    When using DTLSv1_listen() you are using the socket in an unconnected state, so you may receive UDP packets from multiple clients. DTLS is connection based so once DTLSv1_listen() returns successfully you are supposed to create a "connected" socket to the client address. So you have one socket for listening for new connections, and one socket per client communicating with your server.