Fuzzing The New Groupchats

With the new groupchats almost complete I thought it was time for another big testing session, and this time I decided I would take you all along for the roller coaster ride of fun. Before you read any further, I should warn that this is a technical post geared towards the programmers and developers in the crowd. If protocols and C code aren’t your thing, click here.

For those who don’t know what fuzzing is, it’s a form of random code testing. The idea is simple: feed your program random/malformed data and wait for it to crash;  gather crash info, fix the crash, and repeat (or log the crash and keep going). The purpose of this type of testing is to explore the deep dark corners of the state space in order to catch bugs that normal usage or unit testing would almost certainly miss. These types of bugs—particularly in memory unsafe languages such as C—can lead to subtle vulnerabilities that leave the door open for exploitation.

There are different levels of fuzzing ranging from dumb to smart, which explore different aspects and depths of the application protocol. An example of the dumbest form of fuzzing would be to fill a packet with completely random data and send it to the packet handler. Dumb fuzzing theoretically guarantees 100% code coverage given enough time, but in reality the universe will probably end before it ever gets past the first hash check. Conversely, smart fuzzing creates valid data and modifies/corrupts it to varying degrees. This ends up covering a smaller state space, but in practice has much greater code coverage, and is generally more useful for all but the simplest of programs.

I chose only to use two levels, where level-1 is dumb-ish and level-2 is smart-ish. Something highly complex like a compiler or a web browser might require 6 or 7 levels or even more, but the groupchat protocol is pretty simple and more straightforward by comparison. In hindsight, a 3rd level might be warranted, but each added level greatly increases the amount of work, and I’m not convinced that it would be worth the effort, as even level-1 fuzzing has satisfactory code coverage much of the time.

Before going any further, it would be helpful to understand the structure of a Tox groupchat packet:

 [  IP/UDP  |  Header 1  |  Header 2  |  App data  ]

Zooming in to the three sections we care about:

Header 1:   [  Packet ID  (1 b)  |  Chat ID hash  (4 b)  |  Sender PK (32 b)  |  nonce (24 b)  ]

Header 2:  [  Random padding (0-8 b)  |  Group packet type (1 b)  ]

App data:  [  Sender pub-key hash (4 b)  |  payload (0-? b) ]

The first header is a plain-text section comprised of integrity checks; if any of these are invalid the packet will be discarded. The second header, which is encrypted along with the rest of the packet, mitigates certain types of length-based packet analysis with padding, and tells us what higher-level function needs to be done. Due to the very simple nature of these headers and the straightforward way in which they’re handled, unit testing and manual inspection would be better suited for this section of the code. Instead, I skipped all that and focused my efforts on the app data, or more specifically, the payload.

Level-1 targets everything starting after the sender pub-key hash in the app data. The code looks like this. What it does is create a data buffer containing a valid public-key hash and a bunch of random data as the payload (this is the app data section of the packet, which is always created before adding the headers). Then the packet is sent n times to each peer in the group, where n is the number of different packet types that we want to test. If the for loops in fuzz_send_group_packet() confuse you, check out the GROUP_PACKET_TYPE and GROUP_BROADCAST_TYPE enumerators in the group_chats.h file.

A few of the packet types only have a single data field in their payload, so for them this is actually close to the highest possible level of fuzzing. An example of this would be normal text messages, where the entire payload is the message itself. On the other hand, some of the packet types have numerous fields with sensitive parsing operations tied to them, and these are the ones that require us to go a bit deeper. For example, the sync response payload looks like this:

[  num_peers  |  peer_data_1  |  peer_data_2  |  etc.  ]

What makes it complicated is the fact that peer data is itself just another series of data chunks of varied sizes (a protocol within a protocol), all of which must be treated as untrusted and potentially malicious. What happens if num_peers doesn’t reflect the actual number of peers being sent? What happens if critical parts of the peer data gets corrupted, such as the nick length? What happens if the packet is too small?

To answer these questions, we move to level-2 smart-ish fuzzing. I first need to create a valid sync response packet. There are different ways you could do this, but I chose the path of least resistance and copy-pasted most of the code for creating sync responses, adjusting things where needed. Most of that code isn’t too relevant here; the interesting part is the call to fuzz_gc_packet() at the bottom. Go ahead and take a look at what’s inside that function, as it’s the most critical part of the smart-fuzzing code, and differs quite a bit from the way level-1 randomizes packets.

Rather than randomizing every byte in the payload, bytes are randomized with a probability f, called the fuzz-factor, with use of the fuzz_this_byte() function. This is because I want to test packets with different levels of corruption. Having f constantly cycle with clock ticks means I can test payloads with a level of corruption ranging from 0 bytes to the entire thing. In addition to randomizing bytes, I sometimes also modify the data length. This tests bounds checking, although it’s only useful for packet types that don’t have a strict length requirement. For example, we would want to do this for sync responses because the sync response handler only checks the lower bound with the clause: if (length <= sizeof(uint32_t)).

Once I finished the tedious task of re-creating all the message sender functions in fuzz-form and made sure everything was working properly, it was time to try it out. I re-compiled the code with the LLVM address sanitizer enabled, and ran three concurrent instances (that is, three peers in one group, all spamming each other with fuzzed packets). I would then re-start the session every time it crashed or critically misbehaved. Some people might just settle for running it with a debugger attached, but with C there is no guarantee that invalid memory operations will cause a crash. The address sanitizer is a much more reliable tool for this purpose (though it has its own downsides that make debugging more difficult).

I ended up discovering a total of four bugs all within quick succession,  two of which were memory related (fix 1), and two of which were behaviour related (fix 2). After gaining confidence that there were no more ‘easy’ bugs, I left the three instances running overnight while I slept, and found everything running perfectly when I woke up. Eight hours sounds like a lot of time, but in the world of fuzzing it’s only about average. It’s common for people to leave their fuzzers running over the weekend, or even for weeks at a time, although it’s highly context-dependent.

With that said, I think eight hours without a crash indicates a good deal of robustness (that or my fuzzer isn’t very good; hopefully the former). While I still can’t guarantee that my code is perfectly rock solid, and will not stop testing here, I have much more confidence in it now as we get close to merging the new groupchats into Toxcore’s master branch.

If anyone has any suggestions, constructive criticism, or spots any errors, I’d love to hear about it in the comments. Happy Toxing!

Successful Indiegogo Campaign!

We are happy to announce that irungentoo’s personal Indiegogo campaign has been successfully funded!

As you may already know, irungentoo has been running a personal fundraiser, to allow himself to dedicate a month of full time work to Tox development. This fundraiser reached its $5,000 USD goal with 6 hours left on the clock. At the time of writing, a total of $5,785 has been raised by 217 funders.

The Tox Project would like to give a big “Thank You!” to all that donated, as well as to all of Tox’s supporters for making this happen, and helping to raise even more money than irungentoo’s original goal.

irungentoo would like to say to everyone:

Thank you, everyone, for making this fundraiser a success. If you bought a perk please send me the related stuff, it will be put into the repository when I collect most of them. If you donated bitcoin in an amount equal or greater to the value of one of the perks and want to receive that perk, just email me at irungentoo@gmail.com with proof (e.g. a message signed with your bitcoin address key).

Unless something happens (computer stops working, etc.), I will be starting my full time work on toxcore on September 8. I will start by writing toxcore documentation, which will take a long time but is important and might invite more people to contribute to toxcore and make sure that Tox does not die if I get hit by a bus. Thank you for making sure that Tox will become even stronger.


We’ll be sitting down with irungentoo very soon to develop a roadmap for the toxcore library. Once it has been developed, we’ll publish it, and you’ll be able to track development progress as it happens.

Thanks again,

The Tox Project

Sound design!

As I’m sure you are all aware, Tox has grown tremendously in terms of UI and usability, yet client sounds are still somewhat lacking.

We are looking for people to help out with creating sounds for clients (ringtones, notification sounds, etc). No programming knowledge is required. Just join us in #tox-dev on Freenode (here is webchat for those of you without an IRC client).


Tox Dev Talks – #3

(Tox Dev Talks is a series of weekly meetings that bring the Tox developer community together to share ideas, progress, and miscellaneous chatter. They take place Saturdays at 16:00 UTC.)

In this week’s meeting, as usual, we discussed a variety of topics, which have been summarized below.

(Edited 2015/08/30: a few factual corrections were made. Apologies for letting them slip past editing -ak)

  • Binaries
    • Package signing
      • Jenkins will handle it
      • Possibility of making Jenkins push notify on changes
  • Policy
    • #tox-dev
      • We discussed how much off-topic discussion and other noise we are willing to allow on #tox-dev
      • Possibly enabling colors for improved build info
      • nurupo expressed concern of color misuse
    • Social media
      • We reminded the managers of each Tox-related account of their responsibilities
      • Blog
        • Social media will now link to EVERY blog post
        • GSoC mentors should come up with a GSoC blog post
  • Our website
    • installgen2 is working on redesigning tox.chat from scratch, this will take 3-7 days to complete
  • ToxAV
    • Mannol has said there will be more commits this (2015-08-29) weekend
  • New groupchats
    • JFreegman is still waiting on irungentoo to implement the difficult codebase changes
  • Core and client updates
    • qTox
      • Zetok will be tagging qTox issues
    • uTox
      • in-progress UI redesign
      • New groupchats once ToxAV gets merged, and the refactor in that branch gets merged
  • Other developments
    • nurupo is working on fixing bugs and refactoring code in code base of GSoC project he mentored
    • codedust is developing https://github.com/codedust/WebTox – a web based tox client

Tox Dev Talks – #2

(Tox Dev Talks is a series of weekly meetings that bring the Tox developer community together to share ideas, progress, and miscellaneous chatter. They take place Saturdays at 16:00 UTC.)

Tox Dev Talks are back after a week-long hiatus due to scheduling difficulties. As we had irungentoo on hand, we spent a lot of time on core issues. Here’s a bit of what was discussed.

  • Tox core
    • Offline messaging
      • The student who was implementing it disappeared. irungentoo will be taking over if he has time.
    • Friendly names / DNS replacement?
      • No real solution was brought up that would be better than what we have now. That said, DNS will remain optional for clients and not a Tox core feature.
    • irungentoo’s to-do list
      • Improve file transfers.
      • Write proper documentation for the code.
      • In general, fix bugs and/or finish some features that aren’t done.
    • New groupchats
      • Currently stalled due to requiring some significant core changes.
    • A/V revamp
      • Currently waiting on client support and some important bug fixes.
  • Packaging
    • qTox binaries are finally available again (Windows and Linux static)
    • An APT repository will be coming soon.
  • Client news
    • µTox
      • New A/V is complete and waiting on aforementioned bug fixes.
      • grayhatter is currently working to improve the UI.
    • qTox
      • New A/V is almost done, audio works, webcam works, streaming desktop doesn’t work and the button functionality still needs to be implemented.
      • tux3 is no longer dead, and has given access to the main repository to DaSpirit and others while he catches up on things.
    • Antox
      • There is currently a naming dispute with the original author.
        • The current developer, subliun, will continue using the Antox name.
  • Other things worth mentioning
    • As some people aren’t happy with the design of tox.chat, installgen2 is going to redesign the site. He will show what he came up with at the next meeting.

If you want to participate in the next Tox Dev Talk, feel free to drop in next week. We hope to keep a stable schedule, but if that doesn’t happen, we will let you know on the calendar.

Regarding irungentoo’s Indiegogo Campaign

irungentoo (our main toxcore developer) has started a personal fundraising campaign that would allow him to dedicate a month of full time work to Tox development. Although this has already been posted on Twitter, reddit and IRC, we have decided to give it a mention here as well, both in order to support our fellow developer, and to clear up some misunderstandings.

Quite a few people have asked if this is an official Tox fundraising campaign. It is not. This is something that irungentoo is doing on his own, separate from the project; all raised funds will go directly to him to be used however he sees best fit. We mentioned in our previous blog post that we wouldn’t be taking any donations as a project until we have set up a proper organization with an emphasis on transparency and protection of assets. This is still the case, however we also mentioned that individual developers are free to take personal donations, which is what this is.

We apologize if this has caused any confusion, and hope that this has sufficiently cleared up any misunderstandings.


Tox Dev Talks – #1

(If you don’t already know, Tox Dev Talks is a series of weekly meetings that bring the Tox developer community together to share ideas, progress, and miscellaneous chatter. They take place every Saturday at 03:00 UTC.)

This week’s Tox Dev Talk turned out exceptionally well, especially as it was the first one. The primary focus was on mobile issues; here is a summary of what was discussed.

  • How do we deal with doze mode in Android M?
    • Do we use GCM?
  • Is a “passive mode” for toxcore so we can reduce battery and data usage possible?
  • How do we deal with backgrounding restrictions on iOS?
    • Possible solution: use VoIP sockets.
  • No concrete decisions as of now.
  • qTox: as tux3 has been absent for a while, the main repo might be moved to DaSpirit’s fork.

Progress updates and to-do:

  • installgen2 (Web)
    • ToxKek – Fix connection and crash bugs, add mobile frontend design, add avatars, add file transfers, add ToxDNS support, add groupchat support, add memes, and add remote server support. (ToxKek is an early-stage HTML/JS Tox client.)
    • Tox.Party – Add Tox3 support and get https certificate.
    • Tox Wiki – Add missing pages and clean up mess
  • chuongv (iOS)
    • Antidote: Implement video calls for this week.
  • subliun (Android)
    • Antox: working on encrypted profile support, fixing some bugs, and getting ready for av support.
    • New ToxDNS host is toxme.io
  • Impyy (C#/Windows)
    • SharpTox – finish new groupchat bindings, write documentation for both new av api and new groupchat api.
  • oranges (Build infrastructure)
    • Jenkins is up and building libraries, some clients are building.
    • Waiting on client devs for some of the other clients.

We hope to see more of the same kind of constructive discussion, and more developers in attendance next week!

Current Situation

As many of you in the Tox community have likely already heard, a serious situation was brought to our attention which has forced the Tox development team to disassociate itself from the Tox Foundation, along with its sole board member, Sean Qureshi (aka Stqism, aka AlexStraunoff, aka NikolaiToryzin). We learned by Sean’s own admission that he “took a loan against the Tox Foundation”, and used the entirety of the foundation’s funds on personal expenses completely unrelated to the project. He did not inform anyone about his actions prior to taking them, then proceeded to disappear for weeks once we found out, ignoring our attempts to contact him and get an explanation.

The exact amount that he took is unknown due to his having complete control over our finances, but it is in the low-thousands. This fund regrettably included a small amount of donation money, but was primarily made up of money that we received by participating in Google Summer of Code last summer.

First, we want to sincerely apologize to the community and take responsibility. We could not have predicted that something like this would happen, but we certainly could have handled our finances in a more responsible and transparent manner. While our development team consists of many skilled programmers and designers, none of us are experienced in business or financial matters. This led us to put too much trust and power into the hands of a single person, who turned out to be just the sort of person who would take advantage of such a situation. We can blame no one but ourselves for this.

Unfortunately, Sean refuses to take responsibility for what he has done, and seems to carry the attitude that what he did was perfectly acceptable. Despite our having spent a great deal of time and effort trying to engage with him, giving him opportunities to pay us back and redeem himself (which is part of the reason why we have waited this long to make an official post about it), he has shown no remorse for his actions, and continues to hold some of our infrastructure “hostage”. This includes the tox.im, toxme.se, and libtoxcore.so domains. For this reason, we have have also been forced to disassociate ourselves with the aforementioned domains and begin again from scratch with a new domain, tox.chat.

In spite of the damage that has been done—which we do not wish to understate—we’d like to look on the bright side of things and consider this a very expensive lesson learned in project management, and life in general. We’ve lost some money, but we’ve gained a ton of insight. We have also been lucky enough to have a few long-standing members of the community step up and help us out with things like server management, and we should have everything back to normal in a short while, with a stronger and better equipped team than before.

As far as finances go, we are not going to repeat the same mistakes twice. We will not be taking any official donations* until we have set up a proper organization with an emphasis on transparency and protection of assets (more details on this at a future date).

In the mean time, we hope that you will continue to support us, if not financially, then in spirit. Despite all of this drama, we have not lost sight of our vision to provide secure, private communications for everyone. Tox development hasn’t had so much as a hiccup in the midst of all this; our second run at Google Summer of Code is going better than our first, and the number of enthusiastic developers who share our vision continues to grow.

Thank you for your understanding and continued support.

* If you still want to give personal donations to individual developers, most of us have bitcoin wallets or paypal accounts and can be reached in IRC (#tox and #tox-dev @ freenode)