One of the prevailing myths of Austin is that the city is a progressive island in the middle of deep-red Texas. And, to be certain, there are more vegan restaurants and Democratic voters in the capital city than there are in, say, Lubbock. But the idea that things like racism and sexism are unusual in Austin gets proven, time and again, to be a myth without much basis in reality.

The latest example occurred at Austin dining institution Kerbey Lane, and quickly went viral. Sirat Al-Nahi and Leilah Abdennabi, who visited Kerbey Lane’s Guadalupe Street location near UT-Austin’s campus on Sunday morning, both wrote about what happened after the encounter. As Al-Nahi wrote on Facebook:

I have never felt more dehumanized and humiliated than I did this morning. Leilah and I went to go grab breakfast at Kerbey Lane Cafe today. She went to park her car and I went to grab us a table. As I’m waiting, this older man begins watching her park through the window, he begins to insult her parking skills and I stay quiet. Then he says, “She should just go back to Saudi Arabia where she came from.” and nobody said anything. I literally couldn’t believe he said that and I asked him to repeat what he had just said. Maybe I should have let it go, maybe I set myself up for this, but I couldn’t believe what he said and I guess I hoped someone would have the humanity to acknowledge it because their silence hurt just as much as his words. But nobody cared when he repeated it and they didn’t care when he added, “And the same goes for you. What? You have a gun in there? Just go ahead and shoot me.” At this point the management got up and came over and proceeded to seat him in a pathetic attempt to deescalate the situation. They chose to seat the racist guy who just made a girl cry right in from them, rather than us. When we spoke to management, they basically said they were sorry it happened, but they can’t do anything. They couldn’t not seat him. The man is across from and watching this whole thing unfold and Leilah goes to talk to him and he says the same things in front of the entire restaurant and nobody says a single thing. As we turn to leave, Leilah, in tears, says, “Just go [sic] everyone knows we were told very racist things and this restaurant doesn’t feel the need to address it because who cares about us?” And somebody called out, “Nobody.” and we left. Because it was true.

Abdennabi added in a separate post that this is the first time that she had experienced Islamophobia in Austin.

We may not be able to agree on much in this country, but surely a majority of Americans can say that harassing young women based on their appearance is rude, aggressive, inappropriate behavior. But the racist man in the story isn’t the only person in it who fails to live up to whatever stories Austin tells itself about its progressivism.

Kerbey Lane CEO mason Ayer wrote a heartfelt apology on the company’s blog on Sunday night, and his message does a good job of standing up for his restaurant’s team—which did make a mistake in how it handled the situation—and taking responsibility as the leader of the company.

It is not the fault of our store leader that she doesn’t know how to handle a heated, hate-filled assault by one of our guests directed at another.  It’s my fault.  It’s my fault because she hasn’t been trained for this situation.  All of our team members understand that racism and intolerance are unacceptable in any context, but as the leader of our organization it’s my responsibility to ensure that in a situation such as this one our store leaders know how to handle the situation in a way that best ensures the safety of our guests and team members while also ensuring an unwelcome patron is removed from the restaurant.  All store leaders and team members, across all locations, will receive this training moving forward.  While I hope another incident like this never occurs at one of our restaurants, it’s imperative that our store leaders know how to properly handle it.

That’s a fine response, and it seems to have stemmed some of the tide of criticism the store has been receiving on social media. But those who focused their outrage at the restaurant and staff were missing another group who let those women down—all of the patrons who sat there waiting on their breakfast in silence.

The store’s manager made a mistake in rushing to seat a customer who was racist and threatening to the restaurant’s other guests, for sure, but the other customers in the restaurant were in a different situation. Patrons didn’t have to consider how to de-escalate a situation in their workplace. Every customer in the restaurant was empowered to speak up on behalf of the women being harassed, but every one of them opted to sit quietly as it happened.

That’s perhaps to be expected. We all like to imagine that in a hostile situation we’d be prepared to stand up for what we believe is right—but when it comes to it, it’s a lot easier to look uncomfortable and wait for it to pass. All of us, not just Kerbey Lane, are able to create our own policies about what we will or will not tolerate from the people around us. The restaurant’s customers—who, on a Sunday morning across from the University of Texas, were probably a lot of bleary-eyed college students nursing hangovers—all decided that their own personal policy was to let this happen in front of them and pretend that it didn’t.

We spend a lot of time talking about how bystanders should react in given situations—the phrase “bystander intervention” is a part of the discourse around campus sexual assault. It’s also part of the argument for campus carry with the suggestion that the same college students who were at Kerbey Lane on Sunday morning should be armed in case they have to step up in a dangerous situation. But what we tend to leave out of that conversation is how our society isn’t built on bystanders stepping up to confront an uncertain situation as it unfolds before us.

That doesn’t excuse the fact that an entire restaurant full of people sat by as two Muslim women were aggressively harassed. Rather, it’s to note that, even in supposedly progressive Austin, we’re good at finding ways to fail the people around us who need our help.