[apache/incubator-teaclave-sgx-sdk] Encrypted websockets in SGX (#291)

[Claus M]

Hi! 

I am using the embedded websockets crate to connect to a websocket endpoint 
([echo.websocket.org](echo.websocket.org)) in a similar style as the tls client 
example - but with the full tls session inside an encalve (see the code below). 
However, it doesn't seem to work with encryption and the server seems to 
respond with an EOF (i.e. a 0 byte read) after sending the websocket handshake. 
(It works without encryption without issues) 

The components I am using is [embedded 
websockets](https://github.com/ninjasource/embedded-websocket), and ports of 
[rustls](https://github.com/mesalock-linux/rustls), 
[mio](https://github.com/mesalock-linux/mio-sgx) and 
[webpki](https://github.com/mesalock-linux/webpki). Here is the code:

~~~rust

#[no_mangle]
pub extern "C" fn start_thread(fd: c_int, hostname: *const c_char, cert: *const 
c_char) -> sgx_status_t {
    let std_stream = std::net::TcpStream::new(fd).unwrap();
    let name_cast = unsafe { CStr::from_ptr(hostname).to_str() };
    let name = name_cast.unwrap();
    let hostname_ref = webpki::DNSNameRef::try_from_ascii_str(name).unwrap();

    let cert_cast = unsafe { CStr::from_ptr(cert).to_str() };
    let cert = cert_cast.unwrap();
    let cfg = make_config(cert);

    let address = name;
    println!("Connecting to: {} ({})", address, name);
    let mut socket = TcpStream::from_stream(std_stream).unwrap();    

    let mut stream = TlsClient::new(socket, hostname_ref, cfg.clone());
    
    let mut buffer1: [u8; 4000] = [0; 4000];
    let mut buffer2: [u8; 4000] = [0; 4000];
    let mut ws_client = ws::WebSocketClient::new_client(ws::EmptyRng::new());
    
    // initiate a websocket opening handshake
    let websocket_options = WebSocketOptions {
        path: "/",
        host: address,
        origin: "",
        sub_protocols: None,
        additional_headers: None,
    };
    
    let mut state = ConnectionState::Connecting;

    let (len, web_socket_key) = ws_client.client_connect(&websocket_options, 
&mut buffer1).unwrap();

    println!("Sending opening handshake: {} bytes", len);
    stream.write_all(&buffer1[..len]).unwrap();
    let s =  std::str::from_utf8(&buffer1[..len]).unwrap();
    println!("-> {}", s);
    
    let t = thread::spawn(move || {

        let mut poll = mio::Poll::new().unwrap();
        let mut events = mio::Events::with_capacity(32);
        stream.register(&mut poll);
    
        loop {
            poll.poll(&mut events, None).unwrap();
            for ev in events.iter() {
                if let Some(is_readable) = stream.ready(&mut poll, &ev) {
                    state = match state {
                        ConnectionState::Connecting if is_readable => {
                            println!("Connecting::READ");
                            ...
                        }, 
                        ConnectionState::Data if is_readable => {
                            println!("Data::READ");
                            ...
                            ConnectionState::Data
                        },
                        _ => {
                            println!("OTHER");
                            ConnectionState::Data
                        }
                    }
                }
            }
        }
    });
    sgx_status_t::SGX_SUCCESS
}
~~~

Relevant part of TlsClient: (this is from the rustls example)

~~~rust

impl TlsClient {
    pub fn ready(&mut self,
             poll: &mut mio::Poll,
             ev: &mio::event::Event) -> Option<bool> {
        assert_eq!(ev.token(), CLIENT);
        
        let mut interest = Some(false);
        if ev.readiness().is_readable() {
            println!("# TLSClient Read interest");
            self.do_read();
            interest = Some(true);
        }

        if ev.readiness().is_writable() {
            println!("# TLSClient write interest");
            self.do_write();
        }

        if self.is_closed() {
            println!("Connection closed");
            return None;
        }

        self.reregister(poll);
        interest
    }
}

// in do_read() the "closing" is set, as indicated by the println
    pub fn do_read(&mut self) {
       ...
        // If we're ready but there's no data: EOF.
        if rc.unwrap() == 0 {
            println!("EOF");
            self.closing = true;
            self.clean_closure = true;
            return;
        }
...

~~~

Running this code above on an Azure confidential compute machine inside the 
18.04 docker container generates this output:


~~~shell
root@2fd3b8e3f502:~/sgx/samplecode/hello-websocket-rust/bin# ./app
[+] Init Enclave Successful 2!
Connecting to: echo.websocket.org (echo.websocket.org)
Sending opening handshake: 193 bytes
-> GET / HTTP/1.1
Host: echo.websocket.org
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: AAAAAAAAAAAAAAAAAAAAAA==
Origin: 
Sec-WebSocket-Protocol: 
Sec-WebSocket-Version: 13


# TLSClient write interest
OTHER
# TLSClient Read interest
EOF
Connection closed
^C
~~~

As I said, it works perfectly fine without encryption - for which I also don't 
have to use mio, but a raw std::TcpStream - so I think this may be where the 
issue is located. Does anyone else have a pointer as to where to go from here 
to try and debug further? 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-teaclave-sgx-sdk/issues/291