PM's on Touch. and on the Android App are being disabled

Dav

Hello all,

I'm sure you've seen the notice, I just wanted to expand upon what exactly is going on.

I got a report of an issue involving private messages on the android app this morning. It's serious enough that we're taking PM access through our API offline whilst we investigate (this should happen very shortly, Declan and the tech team are working on it right now).

This won't affect the desktop version of the site as it runs off the regular VBulletin code and our own API doesn't interface with that.

I will keep you posted here about the issue and what we're doing to fix it. As it stands, we have had a report from someone saying they were not shown their own inbox via the app. I'm waiting to get a more detailed explanation of what's happened - the PM functionallity has been live in the app for about a week and there have been no reports of anything like this, so as you can imagine, this is getting our most full and serious attention.

We'll keep you posted here on this thread.

Dav

Update on this:

So we figured out what the problem was. We all kinda wish we'd done the lotto and you'll see why when I explain what happened.

There is a software product called Varnish which people who run large websites will be familiar with. Varnish manages caching of content so it can be served out to you more quickly. Requests to the private messaging system are made and stored for a millisecond and then released. What we believe happened that caused us to disable the system was that 2 requests via the Android App at the same millisecond and so the same data was sent out to both requests instead of a fresh request for the data going through. The odds of this happening (right down to the millisecond) are not something I'd like to try and calculate, but suffice it to say, you might understand my lotto comments earlier now.

So having found this issue, the fix was simple, we no longer cache anything that has the word private in the URL, so it can't happen again.

PM functionality has been restored now to touch. and the app. We're sorry for the downtime.

Sarky

... is there anyone who would like to try and calculate the odds? I like really small numbers...

KomradeBishop

That's a very interesting vulnerability (work with exploit testing in games, so like learning about this kind of stuff); I can think of some possible further ways it could be exploited:

If you know the URL of a private forum that you don't have access to, maybe you could view it by spamming requests for the forum URL really fast - and maybe you'll luck-out as a forum member with access is viewing the forum, giving you his cached view of it (or alternatively, you could screw up viewing the forum for other users, by triggering a cached "you don't have permission" page).

I'd imagine viewing forum pages works differently, so probably isn't vulnerable, but still - makes it worth auditing all other types of pages that are cached, to see that nothing else meant to be private, can be leaked.


The odds of the PM bug/exploit or this happening accidentally are pretty low, but the odds go up pretty sharply if someone tries to exploit it deliberately, by spamming requests (it's almost a sure thing then)

Boards.ie: Declan

Just to add, we have released a new version of the app with another lair of security.

The odds of this happening do seem to be incredibly remote. We're confident its been remedied and there is no chance it will happen again, but are keeping a close eye on it just to be on the safe side.

That said, I am going to play the Lotto tomorrow and will avoid thunder and lightening storms for the next while...

Boards.ie: Diego

Hi,

Just to be a little more concrete about the solution. Now we are not caching PMs related requests in Varnish, so chances of this happening again are 0 (like Dav said).

Nothing to worry about guys!

Diego

Gordon

Sarky wrote: »

... is there anyone who would like to try and calculate the odds? I like really small numbers...

As with everything in the world, it's 50/50. It either happens, or it doesn't happen...

Rucking_Fetard

Ya cant say that^^, in the other thread it was impossible to read PMs without login creds, theirs always vulnerabilities.

Shellshocks been sitting there in the open for 25 years and no one spotted it.

Boards.ie: Diego

Rucking_Fetard wrote: »

Ya cant say that^^, in the other thread it was impossible to read PMs without login creds, theirs always vulnerabilities.

Shellshocks been sitting there in the open for 25 years and no one spotted it.

Sure can be some vulnerabilities, but that one cannot happen again

Fr_Dougal

Boards.ie: Declan wrote: »

Just to add, we have released a new version of the app with another lair of security.

Cool...like a Batcave?

TouchingVirus

My memory here might be faulty but was it not the case that Varnish doesn't cache anything if you're logged in? Otherwise there would be chaos with me getting served up other logged in user's pages. The response HTTP headers seem to suggest this:

X-Varnish: 1333105997
X-Varnish-Cache-Result: Miss
X-Varnish-Cacheable: No (Cache-Control=private)

How sure are you that this was the cause of the issue - do your logs go to the millisecond and have you two requests at the exact same time? Have you reproduced the behaviour? Have you contacted the Varnish people to tell them that their software does not obey backend cache headers during the request? There are high traffic sites where it's quite plausible they would get the same page request in the same millisecond, I don't remember reading about this behaviour anywhere.

Sarky

Gordon wrote: »

As with everything in the world, it's 50/50. It either happens, or it doesn't happen...

You were always my favourite.

Boards.ie: Declan

TouchingVirus wrote: »

My memory here might be faulty but was it not the case that Varnish doesn't cache anything if you're logged in? Otherwise there would be chaos with me getting served up other logged in user's pages. The response HTTP headers seem to suggest this:

X-Varnish: 1333105997
X-Varnish-Cache-Result: Miss
X-Varnish-Cacheable: No (Cache-Control=private)

How sure are you that this was the cause of the issue - do your logs go to the millisecond and have you two requests at the exact same time? Have you reproduced the behaviour? Have you contacted the Varnish people to tell them that their software does not obey backend cache headers during the request? There are high traffic sites where it's quite plausible they would get the same page request in the same millisecond, I don't remember reading about this behaviour anywhere.

Hi - that behaviour reflects the current set up. We added a varnish rule to block caching of protected requests to the API. In this particular instance it was caching.

TouchingVirus

Hi Declan,

It's weird being on this side of the table for such discussions. Those HTTP headers were from the main site

If a backend (i.e. the API) reports to Varnish that what it's sending back is not cacheable (Cache-Control=private) then it should not matter if two requests are received for the same URL at the same time. If it does matter and Varnish caches those pages (even for a millisecond) then that's a Varnish bug and should be reported as such. If it was the case that you were not sending back Cache-Control headers from the API to Varnish then you can decide in Varnish whether to cache them in the configuration files based on things like the domain or whatever. Is this what happened?

You may think it doesn't matter but there's a confidence thing here - when it comes to private data on the site I'd like to be sure you absolutely know without doubt what the issue was, have reproduced it and can give users a good enough breakdown of the issue and how you're going to prevent it again. Dav's summary covered the last bit very well, but the rest was a bit wishy washy given that I've been responsible for few varnish configurations in my time.

Boards.ie: Diego

Hi,

Our guess is that there was two different requests in the same second. We could not confirm that because we do not have varnish logs, we only have all the cache misses in the lighthttp logs, but we are fairly sure about that.

The problem occured in the Android app (but touch was vulnerable as well), which uses the API.

In the android app we had a timestamp parameter (in seconds, unfortunately) to make the requests different, but it wasn't enough obviously.

We have changed the app and timestamp is now in milliseconds.

Besides this, what was happening is that Varnish was caching PM related requests (And I agree with you that it should not) but now we don't cache anything for authenticated users anymore.

I hope is a little bit more clear now.

Diego

KomradeBishop

That all makes sense, though (as someone who does security testing) it is a touch worrying that no reproduction of the problem was done, so both the problem and the fix could be verified - might be an idea to revisit that, to do a full repro of the issue, and a test of the fix.

Steve

Debugging sub millisecond events..?

Welcome to my world!

I wouldn't wish it on etc..

GreeBo

Boards.ie: Diego wrote: »

Hi,

Our guess is that there was two different requests in the same second. We could not confirm that because we do not have varnish logs, we only have all the cache misses in the lighthttp logs, but we are fairly sure about that.

The problem occured in the Android app (but touch was vulnerable as well), which uses the API.

In the android app we had a timestamp parameter (in seconds, unfortunately) to make the requests different, but it wasn't enough obviously.

We have changed the app and timestamp is now in milliseconds.

Besides this, what was happening is that Varnish was caching PM related requests (And I agree with you that it should not) but now we don't cache anything for authenticated users anymore.

I hope is a little bit more clear now.

Diego

If we are confident that its the cache/millis issue, surely we can recreate it to confirm?