Outcome Health President Shradha Agarwal, left and CEO Rishi Shah. PHOTO: Outcome Health

Last winter, Goldman Sachs Group Inc. presented Silicon Valley investors with what seemed like a hot opportunity in Chicago.

Advertising company Outcome Health was one of the city’s rising stars, run by a charismatic founder with close ties to local billionaire J.B. Pritzker. Revenue was soaring, profits were impressive, investors were promised a guaranteed return, and Goldman’s involvement gave Outcome the backing of a blue-chip firm.

Those were some of the highlights that led experienced startup investors to put $478.5 million into Outcome last spring at a valuation the company said was $5.5 billion. The investors placed the Outcome bet despite multiple warning signs, a Wall Street Journal examination shows, illustrating how even the savviest investors can gloss over potential issues in pursuit of a big score.

Outcome carried significant debt and had no independent board oversight. Its chief executive and co-founder, Rishi Shah, had publicly discussed how the company’s growth strategy required flawless execution to avoid selling something “that doesn’t exist.” He and his co-founder Shradha Agarwal would also reap an unusually large payout from that investment while maintaining control of the company.

An Outcome spokesman said in a statement: “Throughout the monthslong investment diligence process Outcome Health maintained full transparency, cooperated, and proudly shared the company’s 10-year record of delivering health information to patients. Investors actively reviewed hundreds of documents, interviewed employees and spoke directly with customers and partners about Outcome Health’s success.”

Outcome, which was founded in 2006, puts flat screens and tablets in doctor’s offices and streams pharmaceutical ads to them. Its revenue grew swiftly in the years before the May 2017 funding round, but behind the scenes the company faced issues.

Former employees and advertisers alleged that from 2014 through 2016, some Outcome employees had charged advertisers for ad placements on more screens than the startup had installed, allegations which the Journal reported in October. Outcome had offered free advertising worth tens of millions of dollars in 2017 to make up for shortfalls, people familiar with the arrangements said. Some drug advertisers later suspended their business with the company.

In response to an earlier Journal inquiry, a company spokesman said Outcome had put three employees on paid leave and hired the law firm of former U.S. attorney Dan Webb “to review allegations about certain employees’ conduct” raised internally and by the Journal. The spokesman said Outcome “has always upheld the highest ethical standards.”

Some investors—including funds managed by Goldman, Google parent Alphabet Inc. and Mr. Pritzker’s venture firm—are now suing Outcome and its founders for their money back, claiming they were duped by fraudulent documents that inflated Outcome’s product and financial performance. They have also alleged the company charged for advertising space it didn’t deliver.

Outcome says the fraud allegations are baseless and its founders did nothing wrong. Outcome didn’t make either Mr. Shah or Ms. Agarwal available for comment.

This deal will cause “a lot more soul-searching,” said one investor, whose firm has written off its investment in Outcome. “Clearly investors didn’t do enough diligence.”

Venture-capital investing is inherently risky—most startups fail and few deliver outsize returns. So investors have to weigh potential rewards against possible risks, and be sure they understand both.

One question investors might have asked is how Outcome rapidly surpassed older rivals in a traditionally slow-growing industry despite having raised minimal outside capital. That growth was powered in part by a unique strategy to solve what Mr. Shah described as its “chicken-egg” problem, a strategy he explained came with a key risk.

Outcome needed cash to install screens in doctor’s offices, but without a network of screens it couldn’t attract advertisers to spend money, the company’s founders said in video interviews published online.

Some companies that try to build a business by connecting two sides of a marketplace do so by raising investor capital to seed their market. Outcome had bootstrapped itself, meaning it needed a creative way to get cash to install the screens.

Ms. Agarwal in a 2012 interview at a tech event that can be viewed online described how Outcome solved the problem, saying Mr. Shah “negotiated upfront payments” on advertising deals “to have money in the bank and actually go and do these installations and deliver 200 screens, deliver 500 screens, 2,000 screens, that’s how we kept growing.” In a 2016 interview also available online, Mr. Shah described the strategy as bringing “from the future what we need to fuel the future in the present, which is a concept we’ve recycled.”

The risk of that strategy, which Mr. Shah described in both the 2012 and 2016 interviews, was that Outcome needed to execute with “military” precision. “It required you to sell an idea to both” doctors and advertisers, he said in 2016, “and then you couldn’t mistime it, because you know, that would be fraud, right? If you sell something that doesn’t exist.

“It’s like jumping through one train to the other train and making it out and it’s hard right? We’re not the only ones to do it, there are other chicken-and-egg businesses. But to build a chicken-and-egg business that’s capital intensive without any capital is actually pretty hard. I’m not sure it’s been done.”

Ray Rotolo, who previously ran a media agency that bought advertising space from companies like Outcome, said he hadn’t seen an example of another company in the industry charging for screens that it hadn’t yet installed, but that there would be nothing wrong with such a policy provided clients ultimately get what they pay for.

In a statement, an Outcome spokesman said: “Mr. Shah’s comments were about starting a company with a two-sided market and the importance of delivering for both sets of customers, as explained in the interview.” He added that the founders always stressed “the importance of execution with integrity.”

Outcome projected a bright future when it was fundraising about a year ago—its first time raising money from institutional investors.

The investor presentation reviewed by the Journal claimed that drug ads streamed on Outcome’s screens delivered twice the return on investment of television commercials or print ads. Outcome estimated its 2016 revenue doubled to $131 million thanks to big customers such as GlaxoSmithKline PLC, and its 2014 and 2015 financial statements were audited by a prominent firm, Deloitte, according to people familiar with Outcome’s disclosures. The presentation showed Outcome had wide profit margins and a deep bench of newly hired executives. Deloitte declined to comment.

Outcome also had a hometown hero in Mr. Shah, a 32-year-old Chicago-area native who had won entrepreneur awards for his fast-growing startup. Mr. Shah boasted a powerful ally in Mr. Pritzker, now a candidate for Illinois governor and part of Chicago’s prominent business dynasty.

Mr. Pritzker, 53, had known Mr. Shah previously, in part through the business-leadership group Young Presidents’ Organization, according to people familiar with the group. Mr. Shah serves on the board that oversees 1871, a Chicago tech incubator founded by Mr. Pritzker, and the Pritzker Group added Mr. Shah to its team of senior advisers in 2015. Mr. Shah is no longer involved with the Pritzker Group, but is still on the board of 1871.

Goldman Sachs Investment Partners, a unit of the investment bank’s asset-management division, signed on to lead the Outcome investment after teaming up with the Pritzker Group on a prior deal.

As lead investor, Goldman Sachs directed the due-diligence process to spot potential risks, according to people familiar with the fundraising process. Goldman, which invested $100 million of its clients’ money in Outcome, recruited other investors, these people said.

Outcome came with risks that were evident before investors signed on. The company owed $325 million after taking out a loan to buy a rival in 2016, which raised the risk for investors since creditors would be paid back first in a sale or bankruptcy.

Outcome also didn’t have an independent board to provide oversight and accountability from an outside perspective. The board consisted solely of Mr. Shah and Ms. Agarwal before the investment, according to a person familiar with the matter, and only Goldman would join afterward.

Outcome informed investors before the funding of an October 2016 whistleblower claim by a salesman accusing the company of “committing ongoing fraud” by misleading customers about the size of its network, according to court documents. Outcome told the investors the claims were meritless, according to the documents.

Whatever the signs, investors say they couldn’t have foreseen that Outcome would—as they allege in their lawsuit—alter numbers in case studies to make its ads seem more effective. An Outcome spokesman declined to comment on the allegation that it altered case studies, citing ongoing litigation.

People close to the deal said investors liked the large and growing profits that Outcome stated, and were willing to bet on the company because Outcome sweetened the deal with a unique structure whereby it guaranteed investors a 20% annual return. In most startup equity deals, investors simply buy shares at a set price and hope they will rise in value.

With guaranteed upside, investors agreed to earmark $225 million—nearly half of the equity investment—to pay the founders a dividend, according to the lawsuit.

Two venture investors say they passed on the Outcome deal because of the dividend, the company’s debt burden, and a worry that the return guarantee wouldn’t pay off as expected. One of the investors was also wary of the founder-controlled board.

Write to Rolfe Winkler at rolfe.winkler@wsj.com

Dow Jones & Company, Inc. All Rights Reserved.