Running Experiments in a Real Room

One change at a time, same-night comparisons, hold-out nights, and telling a result from noise without statistics.

Every owner runs experiments, usually without calling them that. You try a new start time, a new special, a new way of greeting people, and you watch what happens. The trouble is the watching. You change three things at once, the weather turns, a holiday lands in the middle, and by the end you have a story and no evidence. A real experiment is the same curiosity with a few rules that stop you from fooling yourself. That isn’t a character flaw. It’s what happens when a busy person tries to learn from a crowded week.

The rules are simple. Change one thing. Write down what you expect before you start. Compare the changed night with a fair comparison, either the same kind of night before, or alternating nights, or a hold-out night that stays the same. Run it long enough to see more than one result. And decide in advance how big a difference would count, so you aren’t deciding afterward when you’re hoping. None of it needs a statistics course. It needs a pencil, patience, and the baseline from Chapter 23. The method in this chapter borrows the logic of a fair test from science and leaves out the formulas. What remains is a habit any owner, teacher, bartender, booker, or musician can keep.

This chapter shows how to run a fair test in a room with real constraints: small numbers, nights that differ, a crowd that talks to itself, and staff who can’t hold perfectly still. We’ll go through the design, the length, how to tell a result from noise using the swing, and how to avoid the common traps. At the end you’ll fill in the Experiment Planner, one page for a single change, and Appendix D carries a copy. It’s the tool that turns Chapters 10 through 20 from a list of ideas into a method of improvement. It also shows which kinds of tests suit a small room and which don’t, so you don’t waste a season on a comparison that could never give a clear answer.

PR TACTIC: The Prediction Line

What it is: Before you start, write one sentence: “If this works, [number] will move from [baseline] to at least [target] within [time].”

Why it works: A prediction made in advance can’t be bent to fit whatever happens, which is the main way small tests go wrong.

Try this: Write the sentence for the next change you plan, date it, and put it where the staff can see it.

31.1 One Change at a Time

The first rule is the hardest to keep. Change one thing. If you alter the start time, the poster, and the cover on the same Thursday, and the night is better, you won’t know which of them helped, or whether the three worked against each other and a fourth thing saved you. In a small room it’s tempting to do everything at once because the problem feels urgent. But a night that improved for unknown reasons can’t be repeated, and a night that worsened for unknown reasons can’t be fixed. Pick the change you believe in most and test it alone. It helps to keep a list of ideas you want to test, in order, and pull the next one off the list only when the current test is finished. The list makes waiting easier, because good ideas aren’t lost, and it makes the order a decision rather than an accident.

Define the change precisely. “Better welcome” isn’t a change. “Every first-timer is greeted, oriented, and introduced to one regular within ninety seconds, by the host, with a card to tick” is. The more exact the description, the more likely staff are to do it the same way each night, and the more likely you are to know whether it was really done. Write the steps on a card. Also decide who does it. A change that depends on whichever staff member happens to remember isn’t a test of the change. It’s a test of memory. Check that the change actually happened. A welcome routine on the planner is a hypothesis. A welcome routine that happened on four of eight nights is a different test. Have the person responsible tick a box each night the change was done, and report the share of nights alongside the result. A tick sheet takes seconds a night and saves the whole test.

Then name the one number the change is meant to move, using the metric tree from Chapter 22, and name the guard. If you’re testing a welcome, the number is the second-visit rate and the guard is the length of the line at the door. Write the baseline next to it, with its swing. You now have the three things every test needs: a change, a number, and a starting point. If you can’t fill all three, you’re not ready to test, and your time is better spent on the baseline. If you have more than one number you care about, pick one as the main measure and treat the others as notes. A test with three equal measures almost always finds something that went up, which is the cherry-picking trap from Chapter 21. If you truly can’t separate two changes, test them together as a package and say so. You’ll learn whether the package works, though not which part did it.

31.2 Fair Comparisons: Before and After, Alternating, and Hold-Out Nights

There are three ways to compare. Before and after is the simplest: measure the night before the change, and again after. It’s fair only if nothing else changed, which is rarely true. Seasons shift, events come and go, and word spreads. Use it when you have a good baseline and a stable period, and write down anything else that happened. Its weakness is that it can’t separate your change from the rest of the world. A before-and-after that shows a gain is encouraging, not conclusive. Before-and-after can be strengthened by looking further back. Instead of comparing one month before with one month after, compare the month after with the same month last year and with the months just before. If the change looks large against all of them, the evidence is stronger.

Alternating is better. If you run the same night every week, apply the change on every other one: welcome, no welcome, welcome, no welcome. Both versions live through the same season, the same weather, the same news. If the changed nights beat the unchanged nights across four or six weeks, the case is much stronger. The cost is a little awkwardness: staff must do the thing on some nights and not on others, and a regular may ask why. For small, internal changes like a greeting or a script, alternating works well. For public ones, like a price, it can confuse guests, and you need another approach. Some changes can’t be alternated because guests would notice or because staff can’t switch habits easily. In those cases, alternate in blocks, such as two weeks with and two weeks without, and accept that the comparison is a little weaker.

A hold-out is a night, or a group, that keeps the old way while everything else changes. If you run a Thursday and a Wednesday and both are similar, apply the change to one and leave the other alone. Or, with a list, send the new message to half of it and the old one to the other half, split at random by something like alternating names. A hold-out controls for the world, because both groups experience it together. It needs enough volume to be meaningful, which is why it suits lists better than small rooms. Whichever method you choose, write it on the planner before you start. Random splitting is simpler than it sounds. Sort a list alphabetically and send version A to every other name, or flip a coin for each name. The point is that you don’t choose who gets which version, because if you choose, your choice may decide the result. Write the method on the planner before you start, and don’t switch methods midway because the early numbers look disappointing.

WHICH COMPARISON FITS WHICH TEST

Before and after: permanent changes that can’t be switched back, such as a new floor or a new name. Weakest evidence. Use the longest baseline you have, and compare with the same period last year.

Alternating nights: staff routines, greetings, lesson changes, set order, scripts. Strong evidence when run for four or more nights each.

Hold-out group: list messages, invitations, offers sent to some people and not others. Strong evidence when the list is large enough and the split is random.

Hold-out night or room: when you run two similar nights or rooms, change one and leave the other. Good evidence if the two really are similar.

31.3 How Long, and How Much

A single night proves nothing, because nights vary. Chapter 23’s swing shows how much. The usual advice for a small room is to run a test for at least four comparable nights on each side, and more if you can. Four against four, eight weeks in all if alternating, is a practical minimum for a weekly night. If your numbers are small, say fifteen guests, a swing of five is a third of the night, and a lot can happen by luck, so run longer. If you can’t run it long enough to see the effect clearly, treat the result as a hint and say so. Seasons matter too. A test that runs across the start of school, a holiday, or a change in the weather can be swamped by the season. Plan tests in stretches where nothing big is expected, and if something big happens anyway, note it and consider repeating the test.

Be careful with outcomes that take time. A thirty-day return can’t be known for thirty days, and a ninety-day return takes three months. Plan around the slowest number you care about. Often the best approach is to use a leading number, like the second-visit rate at fourteen days, to decide early, and then check the lagging one later. Chapter 22 explained the pairing. A test that’s stopped early because it looks good is a common trap: the first few nights are the most likely to be lucky. Fix the length in advance, and don’t peek and stop. If a test must be stopped early for a good reason, such as a safety concern or a clear harm to guests, stop it and write down why. Stopping because the results look bad or good isn’t a good reason. Stopping because people are getting hurt always is.

Don’t forget the cost of a test. Time spent testing is time not spent on something else, and a bad version of a change is run on real guests. Choose tests where the downside is small and reversible. A new greeting is a cheap test. A major price change is an expensive one, and may need a careful plan or a smaller trial. Always write what you’ll do if it works and if it doesn’t. A test with no decision attached is only a curiosity, and a room can’t afford many of those. Write the cost of the test on the planner before you begin: hours, money, and anything given up. If the possible gain is smaller than the cost, don’t run it. If a test would need more nights than you can wait, choose a faster, leading number or a smaller change you can test sooner.

QUESTIONS PEOPLE ASK

“My room is too small to test anything.” It can test changes with big effects, over longer stretches, and it can combine nights into larger groups. It can’t detect small effects reliably, and that’s fine. Focus on changes that might matter a lot.

“What if staff hate the alternating?” Explain why, keep it short, and share the result. Most people tolerate a few weeks of switching once they know the answer will come back to them.

31.4 Result or Noise? Reading It Without Statistics

Statisticians have formal tools for this, and a small room doesn’t need them. A few plain rules go a long way. First, the swing rule: a result counts only if it falls outside the normal swing from your baseline. If your second-visit rate has bounced between 28% and 38% over past cohorts, a rise to 42% is meaningful and a rise to 36% is not. Second, the no-overlap rule for alternating tests: if every changed night beat every unchanged night, that’s strong. If the two sets of results overlap heavily, the change did little. Third, the repeat rule: if the effect shows up twice, in two separate runs, believe it much more. These rules are deliberately cautious. They will sometimes miss a real effect that was too small to see in a small room. That’s an acceptable price. A small room is far more harmed by acting on false results than by missing small real ones, because false results lead to keeping things that don’t work and dropping things that do.

Look at the people, not just the percentages. A rate of 50% on ten people is five people, and one more or fewer moves it ten points. State the counts beside every rate, such as “21 of 42,” so that a reader sees the size of the evidence. And look for a mechanism. If the second-visit rate rose after a change to the welcome, ask whether you can see the welcome working: do guests mention it, do staff notice a difference? A result with a plausible cause, supported by what people say, is more believable than a number alone. Combine the figures with the listening in Chapter 33. Ask staff what they saw as well. A bartender who noticed newcomers lingering longer on welcome nights, or a teacher who saw more students join the floor, is giving you evidence that numbers alone can’t provide. Write their observations on the planner before you look at the counts, so the numbers don’t color what they remember.

Then be humble about what you conclude. “It appeared to help in our room over eight weeks” is a fair sentence. “It increases second visits by 17 points” is a claim that needs more than eight weeks of a small room. If a result surprises you, in either direction, repeat it before acting. If it disappoints you, resist the urge to explain it away, and if it delights you, resist the urge to rush. The most valuable habit in this chapter is the willingness to write “unproven” next to a result you liked. Chapter 21’s test uses that word, and so does Chapter 34’s review. Over time you’ll collect a handful of results you trust. Those few are worth more than any number of confident claims, because you know how each one was earned. Write both the result and the evidence behind it, so the next person can judge it too.

COMMON MISTAKES WITH TESTS

No prediction written first. Results then get bent to fit.

Two changes at once. Credit can’t be assigned.

Stopping when it looks good. Early nights are the luckiest.

Rates without counts. 50% of ten is five people.

Comparing unlike nights. A changed Saturday against an unchanged Thursday proves nothing.

No decision at the end. A test that doesn’t lead to keep, fix, or drop was only a curiosity.

31.5 Common Traps, and What to Do After

A few traps catch almost everyone. Novelty: any change gets a lift because it’s new, and the lift fades. Run long enough to see it. Hawthorne effects: staff behave better when they know they’re being watched, so a test of a staff routine can look better than the routine will be in a normal month. Regression: an unusually good or bad night is usually followed by a more normal one, so don’t read much into a change after an extreme night. Cherry-picking: choosing the comparison after seeing the results. Fixing the plan in advance on the planner prevents it. There’s also a fairness trap. If the people running the changed nights are the best staff and the unchanged nights get whoever is left, the test measures the staff, not the change. Keep staffing as similar as you can on both sides.

After the test, make a decision and write it down. If the change worked, keep it, add it to the standard routine, and update the baseline. If it didn’t, drop it or try a variation, and note what you learned. If it’s unclear, run it again with nothing else changed, or label it unproven and move on. Share the result with the staff and anyone else involved, including the ones who did the work. People are more willing to try the next test when they see the last one led to something. File the planner with the others, so that a year from now you can see what you’ve learned. Decisions should also be proportionate. A small positive result on a cheap change justifies keeping it. A small positive result on an expensive change justifies another test before you commit. A clear negative result justifies stopping, even if you liked the idea.

Over time, the habit is more valuable than any single result. A room that tests one thing a quarter will have learned four things in a year that most rooms never find out: which of their ideas works, which doesn’t, and how much they can trust their own impressions. That’s the real return. It also builds a culture of curiosity rather than blame, because a test that fails teaches as much as one that succeeds. Chapter 34 puts the experiments on a quarterly calendar, and Chapter 39 shows how to stop the ones that aren’t earning their place. The habit also changes how people propose ideas. Instead of arguing about whether something will work, a team that tests can say, “Let’s find out,” and mean it. Keep the planners in a folder with the newest in front. Reading through them once a year is one of the quickest ways to remember what the room has already learned.

TRY THIS WEEK

Choose one change you’ve been meaning to try. Fill in the top half of the Experiment Planner, including the Prediction Line and the comparison method. Show it to the staff, then start on the next comparable night.

CASE STUDY

Welcome, No Welcome, Welcome

Wyatt runs a Thursday at a dancehall and wants to know whether a ninety-second welcome raises the second-visit rate. He alternates for eight Thursdays: welcome on weeks 1, 3, 5, and 7, none on weeks 2, 4, 6, and 8. He writes the prediction first: “If this works, the thirty-day return will rise from about 33% to at least 45%.” The tally of first-timers and those who return within thirty days:

Step 1 — Welcome nights. First-timers 10, 12, 9, 11, returns 5, 6, 4, 6. Totals: 10 + 12 + 9 + 11 = 42 first-timers and 5 + 6 + 4 + 6 = 21 returns. 21 ÷ 42 = 50%. By night: 50%, 50%, 44%, 55%.

Step 2 — No-welcome nights. First-timers 11, 10, 12, 9, returns 3, 4, 4, 3. Totals: 42 first-timers and 14 returns. 14 ÷ 42 = 33%. By night: 27%, 40%, 33%, 33%.

Step 3 — The rules. The difference is 50% − 33% = 17 points, and 21 − 14 = 7 more people on 42 first-timers each side. Swing rule: the old normal was about 28% to 38%, and the welcome nights ran 44% to 55%, entirely above it. No-overlap rule: the lowest welcome night (44%) beats the highest no-welcome night (40%). The prediction of at least 45% overall was met, at 50%.

Step 4 — Humility. Four nights per side is the minimum, and the counts per night are small, so one person moves a night by ten points or more. He writes “appears to help” and repeats the run next quarter. He also asks four guests from welcome nights what they remembered, and three mention being introduced to someone. All figures are illustrative.

REAL TALK

From behind the bar:

I changed my whole posting routine in one week and the numbers jumped. I had no idea why. The next month they fell back. I wasted two months because I couldn’t say what I’d done. Now I change one thing and write it on the fridge.

HEADS UP

Tests that vary prices, promotions, or messages to different guests can raise legal questions. Charging different people different prices or running different offers may involve consumer-protection and discrimination rules. Alcohol promotions must follow Texas Alcoholic Beverage Commission rules (see 16 Texas Administrative Code § 45.103). Splitting a marketing list into groups does not change consent rules for email and text. This is general information, not legal advice. Ask a licensed Texas attorney before testing prices, alcohol promotions, or offers that differ by guest, and see Chapter 35 and Appendix J.

THE EXPERIMENT PLANNER

One page for one change. Fill in the top half before you start and date it. Fill in the bottom half when the test ends.

Fill in: Change being tested, Start date, End date.

• 1. The change Describe it so exactly that anyone could do it the same way. Who does it, and what are the steps?
• 2. The number Which one number should move? Write its definition, its baseline, and its normal swing. Name one guard number.
• 3. The comparison Before and after, alternating nights, or a hold-out? Which nights are in each group, and how many?
• 4. The prediction “If this works, ____ will move from ____ to at least ____ within ____.”
• 5. What else changed List holidays, weather, events, and anything else that could explain a result.
• 6. The result Write the counts and rates for each group. Did the result fall outside the swing? Did the groups overlap? (Result / Unproven / No effect)
• 7. The decision Keep, fix, or drop? What will we do next, and who will tell the staff?

MEASURE IT

The number: Your test count: the number of fair tests you’ve completed in a year, each with a written prediction, a comparison, and a decision.

By hand: Count the planners you’ve filled in to the end: with a prediction written first, a fair comparison, counts for each group, and a decision. Example: 3 completed tests in a year. For any single test, state both counts and rates, such as 21 of 42 against 14 of 42.

Watch out: Novelty fades, so run long enough. Don’t stop early because it looks good, and don’t choose the comparison after you see the results. Label an unclear result unproven.

Go deeper: Chapter 23 (the baseline) and Chapter 34 (the scorecard and the review rhythm).

ROLE BOX

• Owner: Test one change a quarter, such as the welcome, the start time, or the lesson, with the planner filled in first.
• Booker: Alternate two versions of a pitch across similar rooms, and compare reply rates with counts beside the percentages.
• Musician: Test one change to your pitch or your set, such as the opening or the break, over several comparable dates.
• Bartender: Alternate two clip openings across similar posts, and compare sends per reach and guests at the door.
• Dance Teacher: Alternate the lesson’s quick win or the hand-off on alternating weeks, and compare second-lesson rates.

TALK ABOUT IT

• What is the last thing you changed? Could you say whether it worked, and how you’d know?
• Which change would you most like to test? What is the one number it should move?
• What result have you accepted that you never actually tested?